ym88659208ym87991671
JSON Evaluators | Документация для разработчиков

JSON Evaluators

Обновлено 4 марта 2024

Evaluating extraction and function calling applications often comes down to validation that the LLM's string output can be parsed correctly and how it compares to a reference object. The following JSON validators provide functionality to check your model's output consistently.

JsonValidityEvaluator

The JsonValidityEvaluator is designed to check the validity of a JSON string prediction.

Overview:

  • Requires Input?: No
  • Requires Reference?: No
from langchain.evaluation import JsonValidityEvaluator

evaluator = JsonValidityEvaluator()
# Equivalently
# evaluator = load_evaluator("json_validity")
prediction = '{"name": "John", "age": 30, "city": "New York"}'

result = evaluator.evaluate_strings(prediction=prediction)
print(result)
    {'score': 1}
prediction = '{"name": "John", "age": 30, "city": "New York",}'
result = evaluator.evaluate_strings(prediction=prediction)
print(result)
    {'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes: line 1 column 48 (char 47)'}

JsonEqualityEvaluator

The JsonEqualityEvaluator assesses whether a JSON prediction matches a given reference after both are parsed.

Overview:

  • Requires Input?: No
  • Requires Reference?: Yes
from langchain.evaluation import JsonEqualityEvaluator

evaluator = JsonEqualityEvaluator()
# Equivalently
# evaluator = load_evaluator("json_equality")
result = evaluator.evaluate_strings(prediction='{"a": 1}', reference='{"a": 1}')
print(result)
    {'score': True}
result = evaluator.evaluate_strings(prediction='{"a": 1}', reference='{"a": 2}')
print(result)
    {'score': False}

The evaluator also by default lets you provide a dictionary directly

result = evaluator.evaluate_strings(prediction={"a": 1}, reference={"a": 2})
print(result)
    {'score': False}

JsonEditDistanceEvaluator

The JsonEditDistanceEvaluator computes a normalized Damerau-Levenshtein distance between two "canonicalized" JSON strings.

Overview:

  • Requires Input?: No
  • Requires Reference?: Yes
  • Distance Function: Damerau-Levenshtein (by default)

Note: Ensure that rapidfuzz is installed or provide an alternative string_distance function to avoid an ImportError.

from langchain.evaluation import JsonEditDistanceEvaluator

evaluator = JsonEditDistanceEvaluator()
# Equivalently
# evaluator = load_evaluator("json_edit_distance")

result = evaluator.evaluate_strings(
prediction='{"a": 1, "b": 2}', reference='{"a": 1, "b": 3}'
)
print(result)
    {'score': 0.07692307692307693}
# The values are canonicalized prior to comparison
result = evaluator.evaluate_strings(
prediction="""
{
"b": 3,
"a": 1
}""",
reference='{"a": 1, "b": 3}',
)
print(result)
    {'score': 0.0}
# Lists maintain their order, however
result = evaluator.evaluate_strings(
prediction='{"a": [1, 2]}', reference='{"a": [2, 1]}'
)
print(result)
    {'score': 0.18181818181818182}
# You can also pass in objects directly
result = evaluator.evaluate_strings(prediction={"a": 1}, reference={"a": 2})
print(result)
    {'score': 0.14285714285714285}

JsonSchemaEvaluator

The JsonSchemaEvaluator validates a JSON prediction against a provided JSON schema. If the prediction conforms to the schema, it returns a score of True (indicating no errors). Otherwise, it returns a score of 0 (indicating an error).

Overview:

  • Requires Input?: Yes
  • Requires Reference?: Yes (A JSON schema)
  • Score: True (No errors) or False (Error occurred)
from langchain.evaluation import JsonSchemaEvaluator

evaluator = JsonSchemaEvaluator()
# Equivalently
# evaluator = load_evaluator("json_schema_validation")

result = evaluator.evaluate_strings(
prediction='{"name": "John", "age": 30}',
reference={
"type": "object",
"properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
},
)
print(result)
    {'score': True}
result = evaluator.evaluate_strings(
prediction='{"name": "John", "age": 30}',
reference='{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}',
)
print(result)
    {'score': True}
result = evaluator.evaluate_strings(
prediction='{"name": "John", "age": 30}',
reference='{"type": "object", "properties": {"name": {"type": "string"},'
'"age": {"type": "integer", "minimum": 66}}}',
)
print(result)
    {'score': False, 'reasoning': "<ValidationError: '30 is less than the minimum of 66'>"}
ПАО Сбербанк использует cookie для персонализации сервисов и удобства пользователей.
Вы можете запретить сохранение cookie в настройках своего браузера.