cancel
Showing results for 
Search instead for 
Did you mean: 

Any good ways for comparing two floats?

SOLVED
Highlighted
Occasional Contributor

Any good ways for comparing two floats?

HI,

 

I am reading cells from two different excel files and comparing the values in them.

 

For one cell, the values I am comparing are 399.99999998 and 399.99999999.

 

I would like to compare both numbers, and if the difference is greater than 0.0000001, ignore the difference.

 

The problem I am having is, when I get the ABS of num1 - num2, this gives me 1.0000007933E-8.

 

Is there any good way to compare floats without getting E numbers? I am using vbscript.

 

Thanks in advance! 🙂

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Occasional Contributor

I found a solution.

 

I rounded down both numbers to the acceptable decimal places before comparing. 🙂

View solution in original post

2 REPLIES 2
Highlighted
Community Leader

Hi, which programming language do you use?

 

Ahhhhh. I didn't notice you wrote about vbscript.

 

I'm sorry I don't know VB. Maybe someone else could help you.

Highlighted
Occasional Contributor

I found a solution.

 

I rounded down both numbers to the acceptable decimal places before comparing. 🙂

View solution in original post

New Here?
Join us and watch the welcome video:
Announcements
Join TechCorner Challenge!
Want a fun and easy way to learn TestComplete? Try solving weekly TechCorner challenges and get into the Leaderboard!


Challenge Status

Get properties of a web page element

See replies!

Compare images using the Region Checkpoint

See replies!

Compare HTML table with Excel file and correct data in Excel file

Participate!

How to execute remote test and obtain results via Test Runner REST API

Participate!
Top Kudoed Authors