Oil Analysis Accuracy

[Astro14]

Accuracy and precision are two different things and are measured in two different ways. Accuracy (how well shots cluster on a target regardless of location on the target) vs. how close to the shots are to the bullseye (precision). For the lab, you measure precision using standard/certified reference materials. This tells you if 4.5 ppm Fe is really 4.5 ppm Fe - the question here. Accuracy is typically handled through duplicates and ideally, all fo this is done blind so the lab doesn't know which samples are standard or duplicates. So sure, send BS 6 samples of the same oil (is it really the same? How was it collected?) and see how they plot up. In this case, these duplicates are also measuring how well the sampling method represents the oil. You can also have a lab do their own dups where they would take a single sample, homogenize and split it to get duplicate results. Standards would need to be constructed using a new oil with some added anlayte (Fe) in a known concentration and tested. Typically, you have the lab in question run dozens of these standards to develop the mean/SDs to then measure against. You can also "round-robin" these standards out to other labs for comparison obviously assuming the exact same method is used. Labs using equipment like ICP will also have their own internal calibration and QA/QC standards they should be running at some determined interval. I'm sure BS can provide that info if requested like any good lab to give end-users confidence that 4.5 is....4.5.

While I make my living at this point helping clients with questions like this for mineral exploration datasets (I'm an independent consulting geologist) and QA/QC results are a big part of whether I can sign off on their Mineral Resources as a Competent Person so you as investors can be confident that the company has XYZ tons at ABC grade is in fact, as reported. For all the UOAs I've done, I've given zero consideration to QA/QC b/c it's just not that critical for this purpose in my opinion but it raises some questions.

I think you have accuracy and precision backwards, at least as I learned them in Physics.

Precision is the repeatability of result. So that’s the tightness of the grouping in your rifle analogy.

Accuracy is how close that group is the Bullseye, or how close your result is to the true value.

 

[TiGeo]

I disagree with the Wiki definitions; I see that as backwards of what is typically practiced. And it seems (if I have read TiGeo's post correctly), he may agree with me. But if not, it doesn't make either of us wrong. It just is an opportunity to adjust our terms and definitions such that agreement is in place to move forward in discussion.


For the lab geeks out there ... I think this helps us have the conversation:

- Accuracy describes the stdev of the data; the lower the stdev, the more accurate the grouping; smaller is better, generally indicating low variation
- Precision is the definition we use to describe how close that grouping is to the intended target, which is typically an agreed reference standard or desired result
- Calibration is the act of adjusting the precision to move the group towards it's reference standard


What is most important is that there is agreement about what the terms mean, so that consistent conversation can take place.

Not sure why you disagree with the wiki here? That Wiki is likely well vetted. My comments match it unless I'm not reading something correctly? In simple terms accuracy is how close to the target's bullseye you get, precision is how close your grouping is, that's really it. A simple snippet from a preso I use with clients on this:

 

[TiGeo]

I think you have accuracy and precision backwards, at least as I learned them in Physics.

Precision is the repeatability of result. So that’s the tightness of the grouping in your rifle analogy.

Accuracy is how close that group is the Bullseye, or how close your result is to the true value.

Crap, you are right - my bad, mis-typed it.

 

[cvd537]

Again, as far as the labs go, I don't know any of them that have a formal public statement about the accuracy/precision reporting.

We could "assume" that because labs report to the whole number (1,2,3, ...), then they are measuring in the tenths (.1, .2, .3, .....). But without confirmation, it's just a guess.

Feel free to reach out and ask those labs what the ICP reporting threshold means; I think many of us would like to know.

I did reach out, but I did not get anything from them.

Edit: I did get some info from a one analyzer. Typical ICP values are -+15-20% for most elements. Iron for example was 17.5% uncertainty.

 

[TiGeo]

I did reach out, but I did not get anything from them.

Edit: I did get some info from a one analyzer. Typical ICP values are -+15-20% for most elements. Iron for example was 17.5% uncertainty.

So enough to sway typical single-digit analytes in the ppm range up or down ~1 ppm, not a huge amount/enough to be concerned with.

 

[cvd537]

So enough to sway typical single-digit analytes in the ppm range up or down ~1 ppm, not a huge amount/enough to be concerned with.

Agree.

 

[edhackett]

I disagree with the Wiki definitions; I see that as backwards of what is typically practiced. And it seems (if I have read TiGeo's post correctly), he may agree with me. But if not, it doesn't make either of us wrong. It just is an opportunity to adjust our terms and definitions such that agreement is in place to move forward in discussion.


For the lab geeks out there ... I think this helps us have the conversation:

- Accuracy describes the stdev of the data; the lower the stdev, the more accurate the grouping; smaller is better, generally indicating low variation
- Precision is the definition we use to describe how close that grouping is to the intended target, which is typically an agreed reference standard or desired result
- Calibration is the act of adjusting the precision to move the group towards it's reference standard


What is most important is that there is agreement about what the terms mean, so that consistent conversation can take place.

You have it backwards.

Accuracy is how close the lab returns a value to the true value. The only way to determine that is to have the lab's QA/QC data for the run or to have submitted a known value sample along with your unknowns. Without either of those all you can do is make an assumption of accuracy. You know what they say about assumptions.

Precision is how well the lab can replicate a value. It is completely independent of accuracy and good precision says nothing to the accuracy of the value.

Let's say you purchase a certified 100 ppm wear metals in oil standard and submit 5 replicates as blind samples to 3 different labs.

Lab A Fe: 102 103 99 98 106
This data exhibits good accuracy and good precision.

Lab B Fe: 87 85 83 88 87
This data exhibits poor accuracy with good precision.

Lab C Fe: 100 80 110 90 120
This data exhibits good accuracy with poor precision.

Ed

 

[dnewton3]

I still disagree with the terminology from my career experiences, but that really doesn't matter. What matters is concept and context. I have no problem whatsoever adjusting my POV to align with others; that would satisfy the need to have a common ground for the sake of this conversation.

Acknowledging the above, it does not alter the concepts of which we speak. ICP analysis tends to be very "precise". If we accept that a 17.5% uncertainty exists, that still means there's 82.5% certainty; that is not bad at all for machine Repeatability and would certainly contribute to passing a gauge R&R validation.

What most people don't understand is the topic of Stdev and the need for X samples (bare min of 30) with a single controlled variable. The fewer samples taken, the less reliable the calculation is for Stdev; to a point where it becomes parabolically useless. When the sample size is in the single digits, the resulting stdev is approaches being infinitely untrustworthy. See the graph of standard deviation for 95% CI; it represents the difference between the true (real; red and blue lines) stdev value vs the reported value of 1. As you can see, the separation between the reported and true value is huge with low sample sizes. Even at 30 samples, the stdev true value can range from .8 to 1.35, meaning any calculated stdev value still has a considerable amount of range variance, but is at least tolerable. As you approach 100 samples, the slopes of the curves become much, much nearer to parallel (never reaching it; into infinity). And remember, "normal" is defined as 3x the stdev; so take the stdev value and triple it to understand the reality of "normal" variation of any process or product from its average value. IOW, if you took 30 samples, and the stdev result is 1.2, then "normal" variation would be +/- 3.6 ....


When you look at a few UOAs using a lube in your personal use application, there's just no ability to fully understand how much variation is natural to the process or product. Hence, you cannot say with any reasonable certainty whatsoever that one variable is better or worse than another.

 

[cvd537]

Since I started the thread, and it seems to have run its course on the original topic and moved into stats, this is a good read.

https://www.amazon.com/Bernoullis-Fallacy-Statistical-Illogic-Science/dp/0231199945

 

[TiGeo]

I still disagree with the terminology from my career experiences, but that really doesn't matter.

Precision and accuracy as defined by several here (exception my fat-thumbed reversal in my post above) are as they should be. What are you defining differently and how does your career experience change the accepted definitions of these terms?

 

[TiGeo]

You have it backwards.

Accuracy is how close the lab returns a value to the true value. The only way to determine that is to have the lab's QA/QC data for the run or to have submitted a known value sample along with your unknowns. Without either of those all you can do is make an assumption of accuracy. You know what they say about assumptions.

Precision is how well the lab can replicate a value. It is completely independent of accuracy and good precision says nothing to the accuracy of the value.

Let's say you purchase a certified 100 ppm wear metals in oil standard and submit 5 replicates as blind samples to 3 different labs.

Lab A Fe: 102 103 99 98 106
This data exhibits good accuracy and good precision.

Lab B Fe: 87 85 83 88 87
This data exhibits poor accuracy with good precision.

Lab C Fe: 100 80 110 90 120
This data exhibits good accuracy with poor precision.

Ed

Not sure how this is even a discussion, these are the definitions of accuracy/precision.

 

[dnewton3]

Precision and accuracy ... What are you defining differently and how does your career experience change the accepted definitions of these terms?

If you're going to quote me, please quote me in full context ...

I still disagree with the terminology from my career experiences, but that really doesn't matter. What matters is concept and context. I have no problem whatsoever adjusting my POV to align with others; that would satisfy the need to have a common ground ...

For the sake of this conversation, I accept the prevalent definitions most here purport; that allows alignment for the conversation. It's no different than the differences of languages; English vs French vs German, etc ... Each has a word for the color "blue"(English), but as long as they all agree that the word they use represents a wavelength range 450-500 nm, then the conversation can continue. https://sciencenotes.org/visible-light-spectrum-wavelengths-and-colors/

NONE OF THAT really has a bearing on what the OP is concerned about for this thread; that of the results of a UOA. Whether we call it "precision" or "accuracy" does not matter to me; I'm flexible in that regard. More importantly, the underlying question is how reliable is the UOA data in relation to the decisions one makes from that data? Singular (or small sample set) data is utterly useless in micro-analysis. Whereas one can use singular UOA data to compare/contrast it to macro-data with reasonable trust, it is a fool's errand to think a few UOAs are a basis to decide what oil is better or worse than another.

The accuracy and precision of typical ICP results are generally elusive. That the OP got some minimal info is encouraging, given that the process was reported to him as +/- 17.5% (avg). That value would contribute to a decently acceptable gauge R&R study. But that gives me no comfort in the continuing practice most BITOGers have in blindly throwing judgement upon lubes after seeing a few data points. If you consider the fact that the ICP process has some variability in data reporting, and then combine that with the wildly unreliable value of Stdev in small sample sets, they combine to make declarations hollow and essentially useless for the purpose of judging one controlled variable to another.

 

[TiGeo]

For the sake of this conversation, I accept the prevalent definitions most here purport

These definitions are not purported...there is nothing alleged about the definitions of precision and accuracy being used here, they are the standard scientific definitions and have specific meaning that is critical to this conversation. Whether you use a different one in your mind to deal with the difference betweed the two doesn't change the definitions even if we are aligned on how they are being used in this context.

 

[Son_of_shots]

What is the accuracy of an oil analysis? Does anyone have any documentation from Blackstone or SPEEDiagnostix or any other oil analysis lab showing the accuracies for each element tested for, e.g. +- 1 ppm for iron, etc?

A 4 year DAYTIME degree in chemistry is the only way to understand chemical measurement.

 

[aba4430]

This thread from 2010 from a BMW forum appears somewhat relevant to this thread; see post #72.

 

[TiGeo]

This thread from 2010 from a BMW forum appears somewhat relevant to this thread; see post #72.

Good post. Early in my career before I did the exploration/mining deal as a geologist that I do now, I did environmental work and that job effectively was just collecting soil/water samples and sending to a lab then dealing with the data - mostly GCMS data for volatiles and metals. The detection limit bits in that post reminded me of that time in my career and he is correct (reported values would be "<5ppm" for example when it was below the DL. I have a feeling BS will give this info if asked or maybe someone here knows.

 

[cvd537]

A 4 year DAYTIME degree in chemistry is the only way to understand chemical measurement.

Assuming this is facetious and I am missing some background information.

 

[dnewton3]

These definitions are not purported...there is nothing alleged about the definitions of precision and accuracy being used here, they are the standard scientific definitions and have specific meaning that is critical to this conversation. Whether you use a different one in your mind to deal with the difference betweed the two doesn't change the definitions even if we are aligned on how they are being used in this context.

OK. Fine. And ... ???? What's your point? I'm willing to acquiesce to the terms you choose.
You'd rather pick a fight over my willingness to agree to your terms and corresponding definitions, rather than discuss the underlying issue; that of the question of precision/accuracy of an ICP derived UOA?


BTW ... "purport" used in my quote has the intended meaning of "to mean; to signify"; I have no idea why you're Hades-bent on playing word games.
Here is how I used it in the sentence: " ... I accept the prevalent definitions most here purport; that allows alignment for the conversation."
IOW, I agree to use the words and definitions which the majority here present. That allows us to have common ground for context.
Why in the world my agreement and cooperation seems to agitate you is beyond me.

 

[ZeeOSix]

Honestly, send the same sample off twice with a few weeks in between and see how different the results are. I'm betting up to 20%

And also send the same UOA sample to another test lab to compare different lab's results.

 

[ZeeOSix]