DannyAI/embedding_fine_tuning_with_peft_bge_large_en_v1.5

bge-large-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-large-en-v1.5 on the natural-questions dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-large-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • natural-questions
• Language: en
• License: mit

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("DannyAI/embedding_fine_tuning_with_peft_bge_large_en_v1.5")
# Run inference
queries = [
    "can you find a pearl in a mussel",
]
documents = [
    'Freshwater pearl mussel Although the name "freshwater pearl mussel" is often used for this species, other freshwater mussel species can also create pearls and some can also be used as a source of mother of pearl. In fact, most cultured pearls today come from Hyriopsis species in Asia, or Amblema species in North America, both members of the related family Unionidae; pearls are also found within species in the genus Unio.',
    'Ellis Island Generally, those immigrants who were approved spent from two to five hours at Ellis Island. Arrivals were asked 29 questions including name, occupation, and the amount of money carried. It was important to the American government that the new arrivals could support themselves and have money to get started. The average the government wanted the immigrants to have was between 18 and 25 dollars ($600 in 2015 adjusted for inflation). Those with visible health problems or diseases were sent home or held in the island\'s hospital facilities for long periods of time. More than 3,000 would-be immigrants died on Ellis Island while being held in the hospital facilities. Some unskilled workers were rejected because they were considered "likely to become a public charge." About 2% were denied admission to the U.S. and sent back to their countries of origin for reasons such as having a chronic contagious disease, criminal background, or insanity.[43] Ellis Island was sometimes known as "The Island of Tears" or "Heartbreak Island"[44] because of those 2% who were not admitted after the long transatlantic voyage. The Kissing Post is a wooden column outside the Registry Room, where new arrivals were greeted by their relatives and friends, typically with tears, hugs, and kisses.[45][46]',
    "Glee (season 1) The first season of the musical comedy-drama television series Glee originally aired on Fox in the United States. The pilot episode was broadcast as an advanced preview of the series on May 19, 2009, with the remainder of the season airing between September 9, 2009 and June 8, 2010. The season consisted of 22 episodes; the first 13 aired on Wednesdays at 9\xa0pm (ET) and the final 9 aired on Tuesdays at 9\xa0pm (ET). The season was executive produced by Ryan Murphy, Brad Falchuk, and Dante Di Loreto; Murphy's production company helped co-produce the series alongside 20th Century Fox.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7103, 0.3918, 0.2758]])

Evaluation

Metrics

Information Retrieval

• Dataset: NanoQuoraRetrieval
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "query_prompt": "query: ",
      "corpus_prompt": "document: "
  }
  

Metric	Value
cosine_accuracy@1	0.88
cosine_accuracy@3	0.98
cosine_accuracy@5	0.98
cosine_accuracy@10	1.0
cosine_precision@1	0.88
cosine_precision@3	0.4133
cosine_precision@5	0.252
cosine_precision@10	0.14
cosine_recall@1	0.7673
cosine_recall@3	0.952
cosine_recall@5	0.9553
cosine_recall@10	1.0
cosine_ndcg@10	0.9436
cosine_mrr@10	0.9295
cosine_map@100	0.9194

Information Retrieval

• Dataset: NanoQuoraRetrieval
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "query_prompt": "query: ",
      "corpus_prompt": "document: "
  }
  

Metric	Value
cosine_accuracy@1	0.88
cosine_accuracy@3	0.98
cosine_accuracy@5	0.98
cosine_accuracy@10	1.0
cosine_precision@1	0.88
cosine_precision@3	0.4133
cosine_precision@5	0.252
cosine_precision@10	0.14
cosine_recall@1	0.7673
cosine_recall@3	0.952
cosine_recall@5	0.9553
cosine_recall@10	1.0
cosine_ndcg@10	0.9436
cosine_mrr@10	0.9295
cosine_map@100	0.9194

Training Details

Training Dataset

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 80,184 training samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 10 tokens mean: 11.72 tokens max: 24 tokens	min: 11 tokens mean: 132.91 tokens max: 512 tokens

• Samples:

  query	answer
  who wrote i came in like a wrecking ball	Wrecking Ball (Miley Cyrus song) "Wrecking Ball" is a song recorded by American singer Miley Cyrus for her fourth studio album Bangerz (2013). It was released on August 25, 2013, by RCA Records as the album's second single. The song was written by MoZella, Stephan Moccio, Sacha Skarbek, Kiyanu Kim,[2] Lukasz Gottwald, and Henry Russell Walter;[3] production was helmed by the last two. "Wrecking Ball" is a pop ballad which lyrically discusses the deterioration of a relationship.
  what was the purpose of the three-field system	Three-field system The three-field system is a regime of crop rotation that was used in medieval and early-modern Europe. Crop rotation is the practice of growing a series of different types of crops in the same area in sequential seasons. Under this system, the arable land of an estate or village was divided into three large fields: one was planted in the autumn with winter wheat or rye; the second field was planted with other crops such as peas, lentils, or beans; and the third was left fallow, in order to allow the soil of that field to regain its nutrients. With each rotation, the field would be used differently, so that a field would be planted for two out of the three years used, whilst one year it "rested". Previously a "two field system" had been in place, with half the land being left fallow. The three field system allowed farmers to plant more crops and therefore to increase production and legumes have the ability to fix nitrogen and so fertilize the soil. With more crops ava...
  who is the main person in the legislative branch	Article One of the United States Constitution Section 1 is a vesting clause that bestows federal legislative power exclusively to Congress. Similar clauses are found in Articles II and III. The former confers executive power upon the President alone, and the latter grants judicial power solely to the federal judiciary. These three articles create a separation of powers among the three branches of the federal government. This separation of powers, by which each department may exercise only its own constitutional powers and no others,[1][2] is fundamental to the idea of a limited government accountable to the people.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "gather_across_devices": false
  }
  

Evaluation Dataset

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 20,047 evaluation samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 10 tokens mean: 11.79 tokens max: 25 tokens	min: 7 tokens mean: 135.48 tokens max: 512 tokens

• Samples:

  query	answer
  when did call of duty ww2 come out	Call of Duty: WWII Call of Duty: WWII is a first-person shooter video game developed by Sledgehammer Games and published by Activision. It is the fourteenth main installment in the Call of Duty series and was released worldwide on November 3, 2017 for Microsoft Windows, PlayStation 4 and Xbox One. It is the first title in the series to be set primarily during World War II since Call of Duty: World at War in 2008.[2] The game is set in the European theatre, and is centered around a squad in the 1st Infantry Division, following their battles on the Western Front, and set mainly in the historical events of Operation Overlord; the multiplayer expands to different fronts not seen in the campaign.
  who is doing the half time super bowl	Super Bowl LII halftime show The Super Bowl LII Halftime Show (officially known as the Pepsi Super Bowl LII Halftime Show) took place on February 4, 2018 at U.S. Bank Stadium in Minneapolis, Minnesota, as part of Super Bowl LII. Justin Timberlake was the featured performer, as confirmed by the National Football League (NFL) on October 22, 2017.[1] It was televised nationally by NBC.
  when was the sewage system built in london	London sewerage system Joseph Bazalgette, a civil engineer and Chief Engineer of the Metropolitan Board of Works, was given responsibility for the work. He designed an extensive underground sewerage system that diverted waste to the Thames Estuary, downstream of the main centre of population. Six main interceptor sewers, totalling almost 160 km (100 miles) in length, were constructed, some incorporating stretches of London's "lost" rivers. Three of these sewers were north of the river, the southernmost, low-level one being incorporated in the Thames Embankment. The Embankment also allowed new roads, new public gardens, and the Circle line of the London Underground. Victoria Embankment was finally officially opened on 13 July 1870.[3][4]

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• learning_rate: 2e-05
• max_steps: 100
• warmup_ratio: 0.1
• seed: 30
• bf16: True
• load_best_model_at_end: True
• prompts: {'query': 'query: ', 'answer': 'document: '}
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3.0
• max_steps: 100
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 30
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: {'query': 'query: ', 'answer': 'document: '}
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoQuoraRetrieval_cosine_ndcg@10
-1	-1	-	-	0.9583
0.0062	100	0.0156	0.0067	0.9436
-1	-1	-	-	0.9436

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.0
• Transformers: 4.56.1
• PyTorch: 2.8.0+cu126
• Accelerate: 1.10.1
• Datasets: 4.0.0
• Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}