Translation Quality Measurement
By Riccardo Schiaffino and Franco Zearo
Biographical Notes on the Authors
Riccardo Schiaffino
Riccardo Schiaffino worked as translator, translation manager and special
software translation project lead for a major software company. As a translation
manager, Riccardo worked on the improvement of translation quality and on
translation quality metrics and tools. He holds an MA degree in Translation, and
has been working in translations for over 18 years, first in Italy and then in the
U.S. Riccardo is ATA accredited. Contact: [email protected]
Franco Pietro Zearo
Franco Pietro Zearo is a project manager with Lionbridge Technologies in
Boulder, Colorado. He holds a degree in translation from the Advanced School
of Modern Languages for Translators and Interpreters at the University of
Trieste, Italy, and earned an MBA from the University of Phoenix. Before joining
Lionbridge in 1996, he worked as a freelance technical translator in Italian,
English, and Russian. At Lionbridge, he has held positions in translation,
localization analysis, presales, and cultural and globalization consulting. He has
been responsible for translation quality on numerous projects for many Fortune
500 clients. In his previous role as senior technical translator, he helped define
best practices for the translation department. Contact:
[email protected]
Overview
• Technical translation and quality
• Translation quality initiatives
• Quality Control vs. Quality Assurance
• Our proposal for quality assurance
• Checklists
• Sampling techniques
• Conclusions
• Importance of cost/benefit factors
Overview
 Measuring Quality
 Translation Quality Assessment
 Quality Assurance Forms
 Error Categories
 Sampling
 Translation Quality Index
 Questions and Answers
Overview
•
Why Is Quality Measurement Important?
•
•
How to Set Up a Quality Measurement
System
Demo of a Translation Quality Measurement
Tool Prototype
Practical Recommendations
•
Questions & Answers
•
Our Definition of Quality
 Functional approach to quality
 Different
views of translation lead to:
 Different concepts of quality
 Different assessments
 Quality is defined as meeting the needs and
expectations of the customer or user.
Our Definition of Quality
 Functional approach to quality
 Quality is defined as
consistently meeting
the needs and expectations
of the customer or user
Correct Translation
A correct translation is a translation with no
errors or where total error points result in a
Translation Quality Index above the desired
threshold
Customer-driven Considerations

Conformance to specifications
Customer’s vs. One’s own

Fitness for use
How well the translation performs its intended purpose

Value
( = quality & price)
How well the translation performs its intended purpose
at a price customers are willing to pay

Support
E.g.: Printing, testing

Psychological impressions
E.g.: In-country translators; certification
Customer-driven Considerations
Price
Time
Quality
Importance of Quality

Quality as a Competitive Weapon


Good Quality
 Higher Profits
Good quality of
translation (product) and service
(process)
can pay off in higher profits
Improving on quality can reduce costs
and
speed up time-to-market
Cost
Time
Quality
Why is Quality Measurement Important?
 You can’t manage what you can’t measure
 It
is difficult to improve something if you
cannot measure it.
 Such
measurement should be repeatable and
objective.
 Different
persons should arrive at similar
assessment for the same piece of translation.
Why is Quality Measurement Important?
 It
is difficult to improve something if you
cannot measure it.
 Such
measurement should be repeatable
and objective.
 Different
evaluators should arrive at similar
assessment for the same piece of
translation.
Why is Quality Measurement Important?
It is difficult to improve something if you
cannot
measure it
Metrics provide:





A way to objectively quantify a process
A means to reduce the cost of poor quality
A means to increase customer satisfaction
An opportunity for benchmarking
Competitive advantages
“You cannot measure quality”
 This is not true:
 There are certain components of
translation quality that will always remain
subjective.
 However,
 There are other elements that can be
objectively measured.
 By concentrating of these, we believe we
can achieve a satisfactory measurement
of translation quality.
Who Benefits from Reliable Translation Quality
Measurement?
 Professional Translators
 Translation Companies and
In-House Translation Departments
 Translation Customers and Users
Why Do We Make Errors?
 The reasons behind the errors are separate from the
measurement of the errors: Studying why errors
happen is important, but it pertains more to quality
control and improvement than to quality assurance
 E.g., capitalization errors due to the "Autocorrect"
(mis)feature of MS Word (e.g., HBsAg "corrected" to
HbsAg)
QC vs QA
 Quality Control (QC)
 Quality
verification over the whole text.
Example: editing.
 Quality Assurance (QA)
 Sampling
techniques, control of quality over a
(statistically significant) sample of the whole
text.
Example: quality measurement.
QC vs QA
 Quality Control (QC)
 Quality
verification over the whole text.
Example: Editing.
 Quality Assurance (QA)
 Sampling
techniques, control of quality over a
(statistically significant) sample of the whole
text.
Appropriate use: Quality measurement.
Translation Quality Factors
Usability
Legal
Accuracy
Marketing
Inspection Points
Key Principle: Reject “defective material” at its lowest value
Proof
Edit
SL
Content Development
(GIGO)
$ Value of Service
Translation
Stages of Production
Inspection Points
Key Principle: Reject “defective material” at its lowest value
Proof
Edit
SL
Content Development
(GIGO)
$ Value of Service
Translation
Stages of Production
Cost/Benefit Analysis
 Quality
They could help us
identify a
cut-off point.
90
80
70
Time or Money
measurements are a
tool to determine the
optimal level of
quality.
60
50
Investment
40
Value Added
30
20
10
0
1
2
3
Quality Level
4
5
Ideas from other disciplines
 Software project management techniques
 W. Edwards Deming and other quality
assurance experts
When we study translation quality,
we can focus on different things:
The
translator
The translation
process
(the “process”)
Aim:
Feedback
Inputs
Outcomes
Conversion Process
Process
Boundary
Process
Process
Process
Process
Boundary
The translated text
(the “product”)
Product & Process Assessment
Translation quality
assessment must apply
to both:


The translated text
(the “product”)
The translation process
(the “process”)
Aim:
Feedback
Inputs
Outcomes
Conversion Process
Process
Boundary
Process
Process
Process
Process
Boundary
Product & Process Assessment
Translation quality
assessment must apply
to both:


The translated text
(the “product”)
The translation process
(the “process”)
Aim:
Feedback
Inputs
Outcomes
Conversion Process
Process
Boundary
Process
Process
Process
Process
Boundary
Translation Quality Initiatives
The translator
The translation process
DIN 2345
ISO 900x
UNI EN
10754
EUATC
ASTM
ATA and other
translators’
certification
initiatives
SAE J2450
LISA QA
The translated text
Translation Quality Initiatives
 ISO
9002
 EUATC Quality Standard
 DIN 2345
 ASTM Standard for Language Translation
 SAE
J2450
 LISA QA Model
 Academic
translation theories and studies
 Private sector methodologies
Quality Measurement: Our Proposal
 What Can Other Disciplines Teach Us?

Use checklists to collect the data
 Identify types of errors, issues or problems
 Determine relative importance of issues (may be
different for different languages; e.g., spelling
errors in English, French or Italian)
 Use sampling techniques to assess your quality level
 Determine percent thresholds for various levels of
quality
 Determine whether you have achieved your
target quality or not
Criteria for Successful Quality
Measurements
Translation quality measurements should
be:
 Repeatable
(two assessments of the
same sample yield similar results)
 Reproducible
(different evaluators
should arrive at a similar assessment for
the same piece of translation
 Objective
(void of subjective bias)
Classification of Errors
An example of of an error
classification process
(not complete)
Start
End
No
Best detected by
translators
Meaning
Error?
May be detected
by other users
(including
translators)
Yes
Meaning or
form?
Form
Mistranslation
or Addition/
Omissions?
Mistranslation
Misinterpretation of SL
Miswriting of TL
etc.
Grammar or
Style?
Addition/Omission
Style
Grammar
Addition
Omission
etc (?)
Usage or "Style
Proper"?
Syntax
Morphology
Spelling
Punctuation
etc.
Style
Usage
Inconsistency
Non-adherence to
guidelines
etc.
Register
etc.
Measurement through Circumstantial
Evidence
 Errors are circumstantial evidence of
quality
 We believe that precise error
measurement provides sufficient indication
of good and bad translations
 A good translation is a translation with very
few errors or none at all
Definition of Errors
Deal with errors only when they violate agreed upon
protocols of engagement whether implicit or explicit
Examples of explicit and implicit criteria:
 Non-compliance errors (e.g. not following
instructions)
 Violations of generally accepted language
conventions
Summary: Error Categorization
 Select a (small) set of categories

CTQ: Critical-To-Quality categories
 Provide clear definitions
 Set tolerance limits
 Min
/ Max # of errors per X words
 Assign a weight
 Critical,
Major, Minor
Summary: Error Categorization
 Select a (small) set of categories

CTQ: Critical-To-Quality categories
 Provide clear definitions
 Assign a weight
 Critical,
Major, Minor
Real Life Examples
 Development of translation quality
measurement at J.D. Edwards
 Use of sampling techniques for quality
assurance at Lionbridge
The J.D. Edwards’ QA Form
Language Customization

Weighting the major categories
Language Setup
1 - Give appropriate weight to the four following categories (total must add up to 100%)
Accuracy
50%
Style
15%
Categories
Grammar
30%
Formatting
5%
Total 100%
The J.D. Edwards’ QA Form
Language Customization

Weighting the items within the major
categories
2 - Within the Accuracy category, give appropriate weight to the four following items (total must add up to 100%)
Incorrect meaning
40%
Non-standard terminolgy
20%
Accuracy
Inconsistent terminolgy
20%
Untranslated SL
20%
Total 100%
3 - Within the Style category, give appropriate weight to the three following items (total must add up to 100%)
Wrong register
40%
Style
Inappropriate anglicisms
30%
Inappropriate use of passive/active voice
30%
Total 100%
4 - Within the Grammar category, give appropriate weight to the five following items (total must add up to 100%)
Spelling errors
20%
Typos
15%
Grammar
Grammar errors
35%
Syntax errors
25%
Punctuation errors
5%
Total 100%
5 - Within the Formatting category, give appropriate weight to the five following items (total must add up to 100%)
Layout errors
50%
Formatting
Font errors
40%
Double spaces
10%
Total 100%
The J.D. Edwards’ QA Form
Language Customization

Accuracy
Weighting the items within the major
categories (detail)
Incorrect meaning
Non-standard terminolgy
Inconsistent terminolgy
Untranslated SL
40%
20%
20%
20%
Total 100%
How We Worked to Develop Our
Spreadsheet
 Determine type of errors, issues or
problems
 Determine relative importance of issues
(may be different for different languages;
e.g., spelling errors in English, French or
Italian)
 Determine which are the responsibility of
translation
 Determine tolerance limits for various
levels of quality
Translation Quality Measurement Tool
 The Translation Quality Measurement tool
helps to measure process quality
 It is NOT an editing tool, but it serves to
measure whether a process is effective
Use of the Tool
 Use the tool to measure the effectiveness of quality
control process
 Analyze the results obtained through the tool (control
charts)
 If the process is NOT in statistical control
 Discover special causes and deal with them
appropriately
 Remove them if they are negative
 Incorporate them in process if they are positive
 Improve the process when it is in statistical control
A TQI Tool Prototype
ATA Implementation
ATA Implementation
SAE Implementation (Modified)
SAE Implementation (Modified)
TQI Log
Error
Is
Category
Formal
EP
Remarks
Bookmark
Path
File
Grader
Date
x2_is
C:\Documents and
Settings\RS1643403\D
esktop\Quality
Measurement
CoffeMakerTest.
doc
RS164340
3
11/1/200
3
CoffeMakerTest.
doc
RS164340
3
11/1/200
3
2
irregular capitalization; should be is,
not Is
x3_aluminium
C:\Documents and
Settings\RS1643403\D
esktop\Quality
Measurement
x5_food
C:\Documents and
Settings\RS1643403\D
esktop\Quality
Measurement
CoffeMakerTest.
doc
RS164340
3
11/1/200
3
CoffeMakerTest.
doc
RS164340
3
11/1/200
3
aluminium
Formal
1
British spelling; American English
should be aluminum, not aluminium
food
Meaning
2
The container is not made to cook
food, it is made to brew a beverage.
x6_right_for
C:\Documents and
Settings\RS1643403\D
esktop\Quality
Measurement
x7_pyroceram
C:\Documents and
Settings\RS1643403\D
esktop\Quality
Measurement
CoffeMakerTest.
doc
RS164340
3
11/1/200
3
CoffeMakerTest.
doc
RS164340
3
11/1/200
3
CoffeMakerTest.
doc
RS164340
3
11/1/200
3
right for the gas cooker,
the electric plate and the
pyroceram
Formal
1
A better phase might be: "acceptable
for use on gas and electric stoves."
pyroceram
Meaning
4
The word "pyroceram" is unknown to
most English speakers.
x8_wash
C:\Documents and
Settings\RS1643403\D
esktop\Quality
Measurement
x9_trow
C:\Documents and
Settings\RS1643403\D
esktop\Quality
Measurement
wash
Meaning
2
In English, the word "wash" typically
means water and soap. The
instructions specify only using water,
so a better word choice would be
"rinse."
trow
Meaning
2
The word "trow" is a misspelling of
"throw."
total
14
N. of
words
42
TQI
67%
Use of Checklists

There are several quality assessment methodologies that rely on
the use of checklists – among these the LISA methodology.
Q u a lity A s s u ra n c e F o rm
Language:
R e v ie w e r:
D a te :
R e s u lt:
C ritic a l
M a jo r
M in o r
m a x . e rro r p o in ts + 1
5 p o in ts
1 p o in t
C o m m e n ts :
Pass
C lie n t N a m e
P ro je c t N a m e
P ro je c t N u m b e r
P ro je c t M a n a g e r
N u m b e r o f w o rd s
M a x e rro r p o in ts a llo w e d
E rro r C a te g o ry
M is tra n s la tio n
A c c u ra c y
T e rm in o lo g y
Language
S tyle
C o u n try
C o n s is te n c y
0
0
M in o r
M a jo r
m a x . a llo w e d
0
0
0
0
0
0
0
T o ta l
0
M ore elaborate descriptions of the error criteria can be found in the LIS A Q A m odel version 1.0 R eference M anual.
0
0
0
0
0
0
0
C ritic a l
0
0
0
0
0
0
0
to ta l
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Use of Checklists


There are several quality assessment methodologies
that rely on the use of checklists – among these the
LISA methodology.
We would like, however, to advocate the use not of
“universal” checklists, but of checklist specifically
tailored to each language.
 Checklists for evaluating translation companies
 Checklists and tests for evaluating translators
 Checklists for evaluating translations
 Limitations of universal checklists
 Language specific checklists (example, different
weight of spelling correctness for different
languages)
Development of Translation Quality Measurement at
J.D. Edwards
 From the concept of checklists to a spreadsheet of
measurements
 Checklists are appropriate to control whether a
certain action has been performed or not (e.g.,
spell check done or not – as opposed to a
measurement of how many spelling mistakes
were found)
 Based on LISA model (www.lisa.org)
 Flexibility (different settings for different
languages)
Use of Quality Assurance Forms
 The
LISA Quality Assurance Form
Q u a lity A s s u ra n c e F o rm
Language:
R e v ie w e r:
D a te :
R e s u lt:
C ritic a l
M a jo r
M in o r
m a x . e rro r p o in ts + 1
5 p o in ts
1 p o in t
C o m m e n ts :
Pass
C lie n t N a m e
P ro je c t N a m e
P ro je c t N u m b e r
P ro je c t M a n a g e r
N u m b e r o f w o rd s
M a x e rro r p o in ts a llo w e d
E rro r C a te g o ry
M is tra n s la tio n
A c c u ra c y
T e rm in o lo g y
Language
S tyle
C o u n try
C o n s is te n c y
0
0
M in o r
M a jo r
m a x . a llo w e d
0
0
0
0
0
0
0
T o ta l
0
M ore elaborate descriptions of the error criteria can be found in the LIS A Q A m odel version 1.0 R eference M anual.
0
0
0
0
0
0
0
C ritic a l
0
0
0
0
0
0
0
to ta l
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Purposes of sampling according to LISA
 To determine whether something has been
done or not.
 To accept / reject the batch of product at
hand.
 To determine if the process that produced
the product at hand was within acceptable
limits.
Guidelines for Sampling
 Select a sample
 Selection
criteria (e.g. random, systematic)
 Size considerations
 Cost considerations
 Evaluate the sample
 Repeatable,
reproducible, objective
 Investigate the outcome / causes
 Correct / Improve
Statistical Methods
 Defect

Counts
Statistics on Effort Per Defect
 Defect
Density Prediction
 Defect
Pooling
 Defect
Seeding
Defect Counts
 Useful
to obtain a quantitative measurement of
how much QC work to do.
 Ratio
of new defects to defects solved.
 Statistics

on Effort Per Defect
In order to estimate the scope of the defect
correction work, it is necessary to have good data
on the time necessary to fix the various types of
defects
Defect Density Prediction
 One
way to judge whether the QC work on a
translation project is complete is to measure its
defect density (the number of defects per page,
per 1,000 words or per screen).
Defect Pooling
 Defect
pooling is a simple defect prediction
technique that separates the defects found in a
translation sample into two pools.
 Depending
on the number of defects found in
either of the two pools (but not in both) it is
then possible to estimate the defects that have
not been found in the sample.
 This
number can then be used to estimate the
number of defects in the entire project.
Defect Seeding

Defect seeding is a statistical technique in which a
sample of a population is extracted and used to
estimate the total population.

The technique works by deliberately inserting
(“seeding”) defects in a complete translation that will
be QCed.

The ratio of the seeded defects found compared to the
total number of defects seeded provides a rough
estimate of the total number of translation defects yet
to be found.

A common problem with this type of technique is
forgetting to remove the errors deliberately inserted.
Calibration and Error Seeding
One of the things one can do to calibrate a
translation quality measurement tool (or
process) is to use error seeding: Not only to be
able to estimate what percentage of errors is
not discovered, but also in order to estimate
how much variance there is in assessing the
errors that do get discovered.
Suggested process: calibration of a (generic)
translation quality measurement tool
 Have the sample translations (a suitable number of them) scored






"by hand" by expert translators, so as to obtain a suitable range of
evaluated samples, from very good to very bad.
Importance of tightly defining the pool of reviewers
Importance of instructions for reviewers
Have other expert translators score the same tests, but using the
tool
On the basis of the results of the previous two steps, adjust the
weights, types of errors, etc. in the tool until you are satisfied it is
going to help in assessing translation quality - that is, until you
are confident that trained evaluators are going to obtain with the
tool consistent and reliable scores
In doing this remember to remove from the kind of errors that can
be assessed those that are controversial, i.e., those that lead to
differences of opinion whether they are errors or not
Finally adjust the tool so that it produces the range of error scores
that is useful for your organization (e.g., if you want "0" or 100%
as your perfect score)
Translation Quality Index (TQI)
The TQI is a number—obtained by the rigorous
application of a QA process—that indicates the quality
of a given translated text
The concept of a
“Translation Quality Index”
 Translation Quality Index (TQI)
 A number—obtained by the rigorous
application of a QA form—that is indicative of
the quality of a given translation
Delusions of Accuracy
“Averages can be calculated to nineteen places
of decimal with astonishing ease.
When the job is done, it looks very accurate.
It is an easy and fatal step to think that the
accuracy of our arithmetic is equivalent to the
accuracy of our knowledge about the problem in
hand.”
M.J. Moroney, Facts from Figures
Index / Indices
 Depending on one’s purpose, there may
be more than a single TQI.
 E.g., a TQI may be developed for
external purposes (to standardize the
work obtained from outsourcing).
 Another TQI may be primarily for
internal purposes (to measure the
quality of a given special process).
An Example of a
“Translation Quality Index” (1)
 LISA QA Model ver. 1.0 (1995)
 3,000 words (12 pages @ 250 words)
 30 error points
 30 error pts / 3,000 words = 1.0%
 10,000 error pts out of 1 million words
 DPMO = 99.0% = TQI
An Example of a
“Translation Quality Index” (2)
 Microsoft Quality Standards for Print
ver. 1.0 (1998)
 10,000 words (40 pages @ 250 words)
 0 major errors
 15 minor errors
 15 errors / 10,000 words = 0.15%
 1,500 errors out of 1 million words
 DPMO = 99.85% = TQI
An Example of a
“Translation Quality Index” (3)
 2,000 words (8 pages @ 250 words)
 1 critical error
 2 major errors
 3 minor errors
 6 errors / 2,000 words = 0.3%
 3,000 errors out of 1 million words
 DPMO = 99.7% = TQI
Let’s Calculate Two TQIs
LISA QA Model
ver. 1.0 (1995)
ATA Framework for
Standard Error
Marking
3,000 words (12
250 words (estimate)
pages @ 250 words) 17 error points
30 error points
17 error pts / 250
30 error pts / 3,000
words = 0.068
words = 0.01
Implicit TQI = 99.0% Implicit TQI = 93.2%
Control Charts
 Concept of “statistical control”
T Q I C o n tro l C h art
UCL
LCL
M ean
D a t a P o in t
99.25
99.00
98.75
98.50
C 13
C 12
C 11
C 10
C9
C8
C7
C6
C5
C4
C3
C2
Process Flow Diagram
Aim:
Feedback
Inputs
Outcomes
Conversion Process
Process
Boundary
Process
Process
Process
Process
Boundary
Example of Process for Accepting or
Rejecting a Translation Process
1) Determine and describe what your process actually is
(NOT what you think it is or what the process should
be)
2) Measure the quality you have now
3) Determine if you have special cases, and if so,
eliminate them (what the special cases are can be
seen through the use of control charts)
4) Once the process is in statistical control (i.e., any
quality variance is not due to special cases)
5) Change the process to improve quality
6) Measure the new level of quality to determine the
effectiveness of the changes to the process
Very Important
 Improvements made to the overall process
should result in improvements to the product
(the translation)
 Measurements of the product quality should
indicate if there have been actual
improvements to the process
 Therefore, means to measure product quality
must be in place
How to Apply Statistical Methods for Quality Improvement
1.
Define error categories and tolerances
2.
Create a QA form
3.
Obtain a TQI index
4.
Use the TQI index to improve the translation
process
How to Set Up a Quality Measurement System –
Stage 1, Preparation
1. Collect examples of good and bad
translations
2. Analyze the examples to separate
controversial issues from agreed upon errors
3. Decide what to measure (error
categorization)
4. Define what to measure in as many details
as necessary (error definition)
How to Set Up a Quality Measurement
System – Stage 2, Calibration
5. Assign a weight to various types of errors
6. Determine critical errors (if necessary)
7. Repeat 3, 4, 5, and 6 until the system works
in an objective, repeatable, and reproducible
way
Quality Assurance Forms and Tools
Create a QA form (or a tool) to help
graders give objective scores
Q u ality Assu ran ce F o rm
L an g u ag e:
R ev iew er:
D ate:
R esu lt:
C ritical
M ajor
M inor
m ax. error points + 1
5 points
1 point
C o m m en ts:
Pass
C lient N am e
P roject N am e
P roject N um ber
P roject M anager
N um ber of w ords
M ax error points allow ed
E rro r C ateg o ry
M istranslation
A ccuracy
T erm inology
Language
S tyle
C ountry
C onsistency
0
0
M in o r
M ajo r
m ax. allo w ed
0
0
0
0
0
0
0
T o tal
0
M ore elaborate descriptions of the error criteria can be found in the LISA Q A m odel version 1.0 R eference M anual.
0
0
0
0
0
0
0
C ritical
0
0
0
0
0
0
0
3 - Within the Style category, give appropriate weight to the three following items (total must add up to 100%)
Wrong register
40%
Style
Inappropriate anglicisms
30%
Inappropriate use of passive/active voice
30%
Total 100%
to tal
0
0
0
0
0
0
0
2 - Within the Accuracy category, give appropriate weight to the four following items (total must add up to 100%)
Incorrect meaning
40%
Non-standard terminolgy
20%
Accuracy
Inconsistent terminolgy
20%
Untranslated SL
20%
Total 100%
0
0
0
0
0
0
0
0
4 - Within the Grammar category, give appropriate weight to the five following items (total must add up to 100%)
Spelling errors
20%
Typos
15%
Grammar
Grammar errors
35%
Syntax errors
25%
Punctuation errors
5%
Total 100%
5 - Within the Formatting category, give appropriate weight to the five following items (total must add up to 100%)
Layout errors
50%
Formatting Font errors
40%
Double spaces
10%
Total 100%
How to Set Up a Quality Measurement
System – Stage 3, Sampling
 Sampling




Selection criteria (e.g. random, systematic)
Size considerations (the greater the sample,
the more accurate the results)
Select confidence intervals, margins of error
Cost considerations (find the point of
diminishing returns)
How to Set Up a Quality Measurement
System – Stage 4, Measurement
 Measurement

Evaluation must be repeatable, reproducible,
objective

Use of independent auditors

Calculation of a Translation Quality Index
(TQI)
How to Set Up a Quality Measurement
System – Stage 5, Statistical Analysis
Investigate the Outcome
•
At this stage there shouldn’t be any special
causes (use of control charts)
T Q I C o n tr o l C h a r t
UCL
LCL
M ean
D a t a P o in t
99.25
99.00
98.75
98.50
C13
C12
C11
C10
C9
C8
C7
C6
C5
C4
C3
C2
How to Set Up a Quality Measurement
System – Stage 6, Process Improvement
 Take corrective actions (process
improvement)
 Compare the TQI values before and after a
process change to check for actual process
improvement
How to Set Up a Quality Measurement
System – Summary
1. Preparation
2. Calibration
3. Sampling
4. Measurement
5. Statistical Analysis
6. Process Improvement
Practical Recommendations
Importance of
 Glossaries (for terminology)
 Style Guides (for syntax)
 Translation Instructions (for special cases)
 Protocols of Engagement (regulating the treatment
of errors/defects and defining the
acceptance/rejection criteria)
 Translation Guide for Customers (including a
detailed customer checklist to specify what is
important and what is not)
Conclusions
 Desirability of common standards
(see GAAP - Generally Accepted Accounting
Principles)


It is not possible to directly compare different
quality initiatives
A common standard would still permit
assigning different weights to different
categories but in a much more transparent
and comparable way
Translation Quality Scale
Quality Continuum
Translation Quality Scale
Quality Grades
A
100
TQI
B
90
C
80
D
70
E
60
50
Select Bibliography
•Brue, G. : Six Sigma for Managers, New York, McGraw Hill , 2000
•Deming, W. Edwards: Out of the Crisis, Cambridge (Mass), MIT Press, 2000
•Eckersley, H.: “Systems for Evaluating Translation Quality”, in Multilingual Computing & Technology, #47 Volume 13 Issue 3,
April/May 2002
•Grove, A.: High Output Management, 2nd ed., New York, Vintage Press, 1995
•Hönig, H. : “Positions, Power and Practice: Functionalist Approaches and Translation Quality Assessment”, in Schäffner, C.
(ed.) Translation and Quality. Clevendon, Multilingual Matters, 1998
•Language International: “Engineering Language Quality – A word with quality-standards consultant John Gagliardi”, in
Language International Vol. 12 No. 3, June 2000
•Lauscher S.: “Concepts of Translation Quality and Quality Assessment”, in Proceedings of the 39th Annual Conference of the
American Translators Association, 1998
•Ling Koo, S., and Kinds, H.: “A Quality-Assurance Model for Large Projects”, in Sprung, R. (ed.) Translating into Success.
Cutting-edge strategies for going multilingual in a global age. Amsterdam/Philadelphia, John Benjamins Publishing Company,
2000
•LISA: “Microsoft Quality Standards”, in Case Studies and Client Requirements, 1998
•McConnell, S.: Software Project Survival Guide, Redmond, Microsoft Press, 1998
•Moroney, M.J.: “Facts from Figures”, Harmondsworth, Penguins, 1951, 1956(3rd) ,
•Reiss, Katharina: Translation Criticism - The Potential & Limitations. Categories and Criteria for Translation Quality
Assessment. Translated by Erroll F. Rhodes. St. Jerome Publishing 2000
•Schäffner, C. (ed.): Translation and Quality, Clevendon, Multilingual Matters, 1998
•Shewhart, W. : Statistical Method from the Viewpoint of Quality Control. 1939. New York: Dover Publications. Reprint, 1986.
(Originally published: Washington, D.C.: Graduate School of the Department of Agriculture, 1939.)
•Spurr W., and Bonini C. : Statistical Analysis for Business Decisions, Homewood, IL: Richard D. Irwin, Inc., 1967
•Sturz, W.: “DIN 2345 Hits the Language Industry” in Language International Vol. 10 No. 5, May 1998
•Vogel, S.; Nießen, S.; Hermann, N.: “Automatic Extrapolation of Human Assessment of Translation Quality”
http://www-i6.informatik.rwth-aachen.de/PostScript/InterneArbeiten/Paper_2000/Vogel_Evaluation_LREC_2000-corrected.ps.gz
, 2000
•Woyde, R.: “Introduction to the SAE J2450 Translation Quality Metric”, in Language International Vol. 13 No. 2, April 2001
Descargar

Translation Quality Measurement