Benchmarking is crucial for evaluating GPU performance in AI and machine learning. This study measures training speed and scalability across various GPU types to help users choose the best fit for their workloads. Beyond internal assessments, we compare FPT GPU Cloud's performance with similar vendors to highlight key advantages in processing power, memory bandwidth, and scalability. These insights help customers select the most efficient GPU cloud services for their AI needs. Check out the FPT AI Factory’s folk of the Optimum Habana trainer code. These H100 benchmarks may be reproduced by following the provided instructions in that repository. The following benchmarks utilize Habana's Optimum Habana v1.7 trainer code to evaluate the performance of the NVIDIA HGX H100 and HGX H200 against similar vendors. Result H100 (samples per second) – FPT’s Metal Cloud, K8S, DGX, VM. Batch size 54 Model 1 GPU 2 GPUs 3 GPUs 4 GPUs 6 GPUs 8 GPUs Similar Vendor’s H100 80GB SXM 142.3 275 400.6 521.8 740.3 962.2 Compared to 1 GPU (times faster) 1.93 2.82 3.67 5.20 6.76 Metal Cloud – Bare Metal H100 80GB SXM 144.2 283.4 418.9 550.7 799.4 1056.3 Compared to Similar Vendor 101% 103% 105% 106% 108% 110% Compared to 1 GPU (times faster) 1.97 2.91 3.82 5.54 7.33 FPT K8S H100 80GB SXM 143.8 282.4 417.0 546.7 792.8 1046.5 Compared to Similar Vendor 101% 103% 104% 105% 107% 109% Compared to 1 GPU (times faster) 1.96 2.90 3.80 5.51 7.28 DGX H100 80GB SXM 143.8 282.2 417.2 547.7 793.4 1047.0 Compared to Similar Vendor 101% 103% 104% 105% 107% 109% Compared to 1 GPU (times faster) 1.96 2.90 3.81 5.52 7.28 FPT VM H100 80GB SXM (no nvlink) 143.0 261.7 376.6 459.5 Compared to Similar Vendor 101% 95% 94% 88% Compared to 1 GPU (times faster) 1.83 2.63 3.21 Compared to Metal Cloud 99% 92% 90% 83% Result H200 (samples per second) – FPT’s Metal Cloud, multiple batch sizes: 54, 95, 110 Model 1 GPU 2 GPUs 3 GPUs 4 GPUs 6 GPUs 8 GPUs Metal Cloud – Bare Metal H200 141GB SXM (bz54) 158.8 312.4 460.7 600.9 881.4 1165.1 Compared to Similar Vendor’s H100 112% 114% 115% 115% 119% 121% Compared to Metal Cloud H100 110% 110% 110% 109% 110% 110% Compared to Similar Vendor's Baremetal H200 101% 101% 102% 101% 104% 105% Compared to 1 GPU (times faster) 1.84 2.71 3.53 5.18 6.85 Metal Cloud – Bare Metal H200 141GB SXM (bz95) 169.4 332.9 489.2 649.7 917.4 1238.1 Compared to Similar Vendor’s H100 119% 121% 122% 125% 124% 129% Compared to Metal Cloud H100 117% 117% 117% 118% 115% 117% Compared to Similar Vendor’s Baremetal H200 107% 108% 108% 110% 108% 112% Compared to 1 GPU (times faster) 1.96 2.87 3.82 5.39 7.28 Metal Cloud – Bare Metal H200 141GB SXM (bz110) 173.9 341.4 505.8 651.0 973.7 1190.0 Compared to Similar Vendor’s H100 122% 124% 126% 125% 132% 124% Compared to Metal Cloud H100 121% 120% 121% 118% 122% 113% Compared to Similar Vendor’s Baremetal H200 110% 111% 112% 110% 115% 107% Compared to 1 GPU (times faster) 2.01 2.97 3.83 5.72 6.99   FPT AI Factory optimizes GPU performance through advanced infrastructure and software enhancements. Metal Cloud delivers the highest performance across all GPU configurations, outperforming similar vendor benchmarks, with the performance gap increasing as more GPUs are added (up to 110% at 8 GPUs). FPT K8S performs slightly lower due to additional overhead but remains competitive. FPT VM (without NVLink) shows lower performance, especially with multiple GPUs, reinforcing NVLink’s role in scaling efficiency. Across all models, performance scaling is sublinear, with diminishing returns as the number of GPUs increases, though Metal Cloud scales the best (7.33× at 8 GPUs). Meanwhile, the HGX H200, with its larger VRAM (141GB vs. 80GB) and higher memory bandwidth (4.8TB/s vs. 3.35TB/s), enables larger batch sizes and achieves up to 18% better performance than the H100 at maximum batch size. Learn more about FPT AI Factory's services HERE. For more information and consultancy about FPT AI Factory, please contact: Hotline: 1900 638 399 Email: support@fptcloud.com Support: m.me/fptsmartcloud

In the fast-paced world of artificial intelligence, large language models (LLMs) are transforming industries from healthcare to creative writing. However, training these models from scratch can be resource-intensive and impractical for most. That’s where parameter-efficient fine-tuning (PEFT) comes in. It allows you to take a pre-trained model, tweak it for your use case, and do so efficiently — saving time, computing power, and even the planet! By reducing the computational footprint, PEFT makes AI accessible on a wider range of hardware (think laptops or edge devices) and aligns with growing calls for sustainable tech practices by cutting down energy use and carbon emissions. Ready to dive in? Let’s get started! Prerequisites Before you embark on this fine-tuning journey, let’s make sure you’ve got the right tools and setup. Here’s what you’ll need: Hardware: At a minimum, you’ll need 1x NVIDIA A100 80GB GPU to handle PEFT tasks effectively, given the memory and compute demands of models like Llama 3.1–8B. However, in this guide, I’m running the process on Metal Cloud - FPT’s Bare Metal H100 server, which offers even greater power with its NVIDIA H100 GPUs. The H100 is overkill for this tutorial but provides headroom for scaling or handling larger models — expect even faster performance and efficiency compared to the A100. Software: NVIDIA NeMo: Ensure you have NeMo version 24.07 or later installed. You’ll run it via Docker, so familiarize yourself with Docker basics. Python 3.8+: NeMo and its dependencies (like PyTorch and Hugging Face Transformers) require Python 3.8 or higher. Hugging Face CLI or API: For downloading Llama 3.1–8B, you’ll need access to Hugging Face and a valid token. Weights & Biases (WandB): Optional but highly recommended for tracking training progress. Sign up for a free account and grab an API key. NVIDIA Drivers and CUDA Toolkit: Ensure your GPU drivers (minimum version 535 for A100/H100) and CUDA 12.1+ are installed to support NeMo’s GPU-accelerated operations. Access and Permissions: Request access to the Llama 3.1–8B model on Hugging Face, as it requires approval from Meta. These prerequisites set you up for success, whether you’re on a standard A100 or leveraging the H100 server. Let’s dive into the steps! Understanding PEFT: What It Is and Why It Matters Before we jump into the how-to, let’s unpack what Parameter-Efficient Fine-Tuning (PEFT) really means. Picture a massive pre-trained model like Llama 3.1–8B, packed with 8 billion parameters — tiny knobs and dials that define its behavior. This model has already soaked up general knowledge from huge datasets, but now you want it to excel at something specific, like generating Japanese creative writing or answering technical questions. Fully fine-tuning all 8 billion parameters would be like rebuilding a car engine to tweak its radio — it works, but it’s overkill and burns through resources. PEFT flips the script. Instead of adjusting every parameter, it freezes most of the model and tweaks only a small subset — sometimes as little as 0.1% of the total parameters. This keeps the heavy lifting (and GPU memory) to a minimum while still adapting the model to your task. Think of it as adding a custom filter to a pre-built camera lens: you get sharp results without redesigning the whole system. The benefits? Faster training, lower memory use, and the ability to fine-tune on a single GPU — or even a laptop in some cases. Plus, it’s kinder to the environment, slashing the energy cost of AI development. Key PEFT Methods and Parameters PEFT isn’t one-size-fits-all — it comes in flavors, each with its own tricks: LoRA (Low-Rank Adaptation): This method adds small, trainable “adapter” matrices to specific layers (like attention mechanisms). In our example, we target attention_qkv (query, key, value in the transformer). Parameters here include: Rank (r): Controls the size of the adapter matrices. A lower rank (e.g., 8 or 16) means fewer parameters to train, balancing efficiency and expressiveness. Target Modules: Which parts of the model get adapters (e.g., attention layers). More targets = more flexibility, but also more compute. P-Tuning: Instead of tweaking weights, P-Tuning optimizes a set of “prompt” embeddings fed into the model. Key parameter: Prompt Length: How many tunable tokens to add (e.g., 20). Longer prompts can capture more context but increase complexity. Adapter Tuning: Adds lightweight neural layers inside the model. Parameters include: Adapter Size: The number of neurons in these layers (e.g., 64). Smaller sizes keep things light. In our guide, we’ll use LoRA because it strikes a great balance between performance and efficiency, but NeMo supports other methods too — experiment to find your favorite! Hardware Requirements: What You’ll Need To follow along, you’ll need some decent hardware. At a minimum, I recommend 1x NVIDIA A100 80GB GPU for PEFT tasks. Why? The A100’s massive memory and computing power are ideal for handling the tensor operations and parallel processing that NeMo leverages. If you’re on a budget, a smaller GPU like an RTX 3090 (24GB) might work for lighter models, but expect longer training times and potential memory constraints. For optimal performance, especially with larger models like Llama 3.1–8B, stick with the A100 or equivalent. Step 1: Downloading the Llama 3.1–8B Model We’ll kick things off by grabbing the Llama 3.1–8B model in Hugging Face format. This 8-billion-parameter beast from Meta AI is a fantastic starting point for fine-tuning, offering a balance of performance and efficiency. How to Download First, request download permission from Meta’s Hugging Face page (you’ll need to sign up and agree to their terms). Once approved, create a directory to store the model: [code lang="js"] mkdir llama31-8b-hf [/code] You’ve got two options to download: Option 1: CLI Tool Log in to Hugging Face and use their CLI: [code lang="js"] huggingface-cli login huggingface-cli download meta-llama/Llama-3.1-8B --local-dir llama31-8b-hf [/code] Option 2: Python API If you prefer scripting, use this Python snippet (replace <YOUR HF TOKEN> with your Hugging Face token): [code lang="js"] from huggingface_hub import snapshot_download [/code] [code lang="js"] snapshot_download( &amp;nbsp;&amp;nbsp; repo_id="meta-llama/Llama-3.1-8B", &amp;nbsp;&amp;nbsp; local_dir="llama31-8b-hf", &amp;nbsp;&amp;nbsp; local_dir_use_symlinks=False, &amp;nbsp;&amp;nbsp; token="&amp;lt;YOUR HF TOKEN&amp;gt;" ) [/code] Once complete, your model files will land in ./llama31-8b-hf. Pro tip: Verify the download by checking for key files like pytorch_model.bin or model.safetensors—this ensures you’ve got everything intact. Step 2: Converting to NeMo Format NeMo uses its own .nemo format for models, which supports distributed checkpointing and flexible parallelism. Let’s convert our Hugging Face model to .nemo. Launch the NeMo Container Fire up NVIDIA’s NeMo Docker container with GPU support: [code lang="js"] docker run --gpus device=1 --shm-size=2g --net=host --ulimit memlock=-1 --rm -it -v ${PWD}:/workspace -v ${PWD}/results:/results nvcr.io/nvidia/nemo:24.07 bash [/code] This command maps your current directory to /workspace in the container and sets up GPU access. Run the Conversion Inside the container, execute: [code lang="js"] python3 /opt/NeMo/scripts/checkpoint_converters/convert_llama_hf_to_nemo.py --input_name_or_path=./llama31-8b-hf/ --output_path=llama31-8b.nemo [/code] The resulting llama31-8b.nemo file is ready for fine-tuning and supports any tensor parallel (TP) or pipeline parallel (PP) configuration without additional tweaking. This flexibility is a huge win for scaling across multiple GPUs if you expand your setup later! Preparing Your Data Data is the lifeblood of fine-tuning. For this guide, we’ll use the Databricks Dolly 15k Japanese dataset (a translated version of Dolly 15k) as an example, but you can swap in any dataset relevant to your task — think medical QA, customer support logs, or creative writing prompts. Step 1: Download the Dataset Let’s pull the dataset from Hugging Face: [code lang="js"] # load_dataset.py from datasets import load_dataset # Load dataset ds = load_dataset("llm-jp/databricks-dolly-15k-ja") df = ds["train"].data.to_pandas() df.to_json("databricks-dolly-15k-ja.jsonl", orient="records", lines=True) [/code] This saves the dataset as a .jsonl file, where each line is a JSON object with fields like instruction, context, and response. Step 2: Preprocess the Data We need to format the data into a structure NeMo can digest. Here’s a preprocessing script to combine instruction and context into an input field, paired with an output response: [code lang="js"] # preprocess.py import json import argparse import numpy as np def to_jsonl(path_to_data):    print("Preprocessing data to jsonl format...")    output_path = f"{path_to_data.split('.')[0]}-output.jsonl"    with open(path_to_data, "r") as f, open(output_path, "w") as g:        for line in f:            line = json.loads(line)            context = line["context"].strip()            instruction = line["instruction"].strip()            if context:                # Randomize order of context and instruction for variety                context_first = np.random.randint(0, 2) == 0                input_text = f"{context}\\\\n\\\\n{instruction}" if context_first else f"{instruction}\\\\n\\\\n{context}"            else:                input_text = instruction            output = line["response"]            g.write(                json.dumps(                    {"input": input_text, "output": output, "category": line["category"]},                    ensure_ascii=False                ) + "\\\\n"            )    print(f"Data saved to {output_path}") def get_args():    parser = argparse.ArgumentParser()    parser.add_argument("--input", type=str, required=True, help="Path to jsonl dataset")    return parser.parse_args() if __name__ == "__main__":    args = get_args()    to_jsonl(args.input) [/code] Run it like this: [code lang="js"] python preprocess.py --input=databricks-dolly-15k-ja.jsonl [/code] Step 3: Split the Dataset Now, split the preprocessed data into training, validation, and test sets: [code lang="js"] # split_train_val.py import json import random input_file = "databricks-dolly-15k-ja-output.jsonl" train_file = "training.jsonl" val_file = "validation.jsonl" test_file = "test.jsonl" train_prop, val_prop, test_prop = 0.80, 0.15, 0.05 with open(input_file, "r") as f: lines = f.readlines() random.shuffle(lines) total = len(lines) train_idx = int(total * train_prop) val_idx = int(total * val_prop) train_data = lines[:train_idx] val_data = lines[train_idx:train_idx + val_idx] test_data = lines[train_idx + val_idx:] for data, filename in [(train_data, train_file), (val_data, val_file), (test_data, test_file)]: with open(filename, "w") as f: for line in data: f.write(line.strip() + "\\\\n") [/code] This gives you three files: training.jsonl (80%), validation.jsonl (15%), and test.jsonl (5%). Here’s a sample of what the processed data looks like: [code lang="js"] { "input": "若い頃にもっと時間をかけてやっておけばよかったと思うことは？", "output": "健康とウェルネスへの投資だ。若い頃に運動やバランスの取れた食事、家族との時間をもっと大切にしていれば、今後の人生がもっと豊かで楽になっていただろう。", "category": "creative_writing" } [/code] Step 3: Fine-Tuning with PEFT Time to fine-tune! We’ll use the LoRA method (as set in PEFT_SCHEME="lora"), though you can switch to P-Tuning or others by tweaking that variable. Here’s the full script: [code lang="js"] MODEL="llama31-8b.nemo" TRAIN_DS="[training.jsonl]" VALID_DS="[validation.jsonl]" TEST_DS="[test.jsonl]" TEST_NAMES="[data]" PEFT_SCHEME="lora" CONCAT_SAMPLING_PROBS="[1.0]" TP_SIZE=1 PP_SIZE=1 huggingface-cli login --token <HF_TOKEN> export WANDB_API_KEY=<WANDB_TOKEN> wandb login torchrun --nproc_per_node=1 \\\\ /opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py \\\\    trainer.devices=1 \\\\    trainer.num_nodes=1 \\\\    trainer.precision=bf16 \\\\    trainer.val_check_interval=20 \\\\    trainer.max_steps=50 \\\\    model.megatron_amp_O2=True \\\\    ++model.mcore_gpt=True \\\\    ++model.flash_attention=True \\\\    model.tensor_model_parallel_size=${TP_SIZE} \\\\    model.pipeline_model_parallel_size=${PP_SIZE} \\\\    model.micro_batch_size=1 \\\\    model.global_batch_size=32 \\\\    model.optim.lr=1e-4 \\\\    model.restore_from_path=${MODEL} \\\\    model.data.train_ds.file_names=${TRAIN_DS} \\\\    model.data.train_ds.concat_sampling_probabilities=${CONCAT_SAMPLING_PROBS} \\\\    model.data.validation_ds.file_names=${VALID_DS} \\\\    model.peft.peft_scheme=${PEFT_SCHEME} \\\\    model.peft.lora_tuning.target_modules=[attention_qkv] \\\\    exp_manager.create_wandb_logger=True \\\\    exp_manager.explicit_log_dir=/results \\\\    exp_manager.wandb_logger_kwargs.project=peft_run \\\\    exp_manager.wandb_logger_kwargs.name=peft_llama31_8b \\\\    exp_manager.create_checkpoint_callback=True \\\\    exp_manager.checkpoint_callback_params.monitor=validation_loss [/code] Key Highlights LoRA in Action: We’re targeting attention_qkv modules, adding small adapters to fine-tune efficiently. WandB: Tracks training progress — super handy for visualizing loss curves. Precision: Uses bf16 (bfloat16) for faster training with minimal accuracy loss on modern GPUs. Adjust max_steps (how many training iterations) or global_batch_size (how many samples per update) based on your dataset size and hardware. For our small example, 50 steps keep things quick. Diving Deeper: Understanding the Parameters Want to geek out on what’s driving this PEFT fine-tuning? Here’s a quick rundown of the most important parameters in the script and why they matter for keeping Llama 3.1–8B manageable on a single A100 — or, in my case, FPT’s bare metal H100 server: trainer.precision=bf16: Uses bfloat16 precision for faster, memory-efficient training on modern GPUs like the A100 or H100. It’s a PEFT superpower, slashing memory use while keeping accuracy sharp. trainer.max_steps=50: Limits training to 50 steps, keeping things quick for small datasets like ours. Bump this up for larger data or better results, but watch for longer runtimes. model.micro_batch_size=1 & model.global_batch_size=32: Sets the batch size per GPU (1 sample) and total batch size (32 samples across GPUs). Low micro-batch size saves memory for PEFT, but you might tweak it higher if your GPU (like the H100’s 94GB) can handle more. model.optim.lr=1e-4: Sets the learning rate to 0.0001, a small value ideal for PEFT’s delicate parameter updates (like LoRA adapters) to avoid overshooting. model.peft.peft_scheme=lora & model.peft.lora_tuning.target_modules=[attention_qkv]: Uses LoRA for efficiency, targeting only the attention query, key, and value layers. This keeps parameter updates minimal—perfect for resource-light fine-tuning on high-performance hardware like the H100. exp_manager.create_wandb_logger=True: Enables Weights & Biases logging to track progress live. It’s your window into loss curves and resource use, making it easier to tweak and troubleshoot, especially on a powerful setup like Metal Cloud- FPT’s H100 server. These parameters work together to make PEFT fast, efficient, and scalable. Tweak them based on your hardware, dataset, or goals — PEFT’s flexibility is one of its biggest perks! Visualizing PEFT Performance: Resource Usage During Fine-Tuning Curious about what’s happening under the hood during PEFT fine-tuning? Check out this snapshot of resource metrics from the fine-tuning process (see the graphs below). These charts, captured over 500 seconds, show how our Llama 3.1–8B model behaves on an NVIDIA A100 GPU: Memory Usage: System memory stays low (peaking at ~1.8%), while GPU memory ramps up to ~26GB (or 34–40% of the A100’s 80GB), reflecting the memory demands of loading and processing the 8-billion-parameter model and its PEFT adapters. GPU Power and Utilization: The GPU draws up to 500W and operates at 80–85% utilization, showcasing the A100’s efficiency in handling the tensor operations and parallel processing in NeMo. This confirms PEFT’s promise of staying resource-light compared to full fine-tuning. Memory Access Time: GPU time spent accessing memory hovers around 30–40%, indicating balanced compute and memory operations — ideal for PEFT’s low-parameter adjustments. These metrics highlight why PEFT is a game-changer: It keeps resource usage manageable, even for a hefty model like Llama 3.1–8B, making fine-tuning feasible on a single high-end GPU. If you’re tweaking hyperparameters or scaling up, expect these patterns to shift — play around and monitor your own runs for insights! Step 4: Running Inference Finally, let’s test our fine-tuned model! This script evaluates performance on the test set: [code lang="js"] MODEL="llama31-8b.nemo" PATH_TO_TRAINED_MODEL="/results/llama31-8b_lora.nemo"  # Adjust based on output from training TEST_DS="[test.jsonl]" TEST_NAMES="[data]" OUTPUT_PREFIX="./results/peft_results" TP_SIZE=1 PP_SIZE=1 [ ! -d ${OUTPUT_PREFIX} ] && mkdir -p ${OUTPUT_PREFIX} python3 \\\\ /opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_generate.py \\\\    model.restore_from_path=${MODEL} \\\\    model.peft.restore_from_path=${PATH_TO_TRAINED_MODEL} \\\\    trainer.devices=1 \\\\    model.tensor_model_parallel_size=${TP_SIZE} \\\\    model.pipeline_model_parallel_size=${PP_SIZE} \\\\    model.data.test_ds.file_names=${TEST_DS} \\\\    model.data.test_ds.names=${TEST_NAMES} \\\\    model.global_batch_size=32 \\\\    model.micro_batch_size=4 \\\\    model.data.test_ds.tokens_to_generate=20 \\\\    inference.greedy=True \\\\    model.data.test_ds.output_file_path_prefix=${OUTPUT_PREFIX} \\\\    model.data.test_ds.write_predictions_to_file=True [/code] This generates responses for your test inputs and saves them to [code lang="js"] ./results/peft_results_data_preds_labels.jsonl. [/code] Dive into the output to see how your model performs—did it nail those Japanese creative writing prompts? Wrapping Up And there you have it — a complete guide to fine-tuning Llama 3.1–8B with FPT AI Factory, NVIDIA NeMo, and PEFT! From understanding the magic of parameter-efficient methods to running inference, you’ve now got the tools to adapt LLMs to your own projects. Play around with different datasets, tweak LoRA’s rank, or scale up to multiple GPUs — the possibilities are endless. For more information and consultancy about FPT AI Factory, please contact: Hotline: 1900 638 399 Email: support@fptcloud.com Support: m.me/fptsmartcloud Source: https://blog.usee.ai/a-step-by-step-guide-to-fine-tuning-models-with-nvidias-nemo-framework-49ba3ab27d3d

1. Overview This guide provides a comprehensive walkthrough for setting up and running distributed training using LLaMA-Factory on Metal Cloud (Bare Metal Server). We cover environment setup, Slurm-based job scheduling, and performance optimizations. Additionally, we include a training task using the Open Instruct Uncensored Alpaca dataset, which consists of instruction-tuning samples, for fine-tuning a Llama-3.1-8B model, with full fine-tuning settings on 4 nodes, 8 x NVIDIA H100 GPUs per node, providing hands-on instructions for replicating a real-world training scenario. The execution setting up a distributed training environment on Metal Cloud using Slurm, an open-source workload manager optimized for high-performance computing. The guide walks through: Preparing the infrastructure with Slurm, CUDA, and NCCL for efficient multi-GPU communication. Installing LLaMA-Factory and configuring the system to enable seamless model training. Running a fine-tuning task for the LLaMA-3.1-8B model using the Open Instruct Uncensored Alpaca dataset. Leveraging Slurm’s job scheduling capabilities to allocate resources and monitor performance efficiently. Key highlights that readers should focus on: Scalability & Efficiency: The guide demonstrates how Slurm optimally distributes workloads across multiple GPUs and nodes, reducing training time. Cost Optimization: Proper job scheduling minimizes idle GPU time, leading to better resource utilization and lower costs. Reliability: Automated job resumption, error handling, and real-time system monitoring ensure stable training execution. Hands-on Training Example: A real-world fine-tuning scenario is provided, including dataset preparation, YAML-based configuration, and Slurm batch scripting for execution. By following this guide, readers can replicate the training pipeline and optimize their own LLM training workflows on Metal Cloud. 2. Why Slurm for Distributed Training? Slurm is a widely used open-source workload manager designed for high-performance computing (HPC) environments. It provides efficient job scheduling, resource allocation, and scalability, making it an excellent choice for AI training on Metal Cloud. Key advantages include: Resource Efficiency: Slurm optimally distributes workloads across GPUs and nodes, minimizing idle resources. Scalability: Seamlessly scales from a few GPUs to thousands, accommodating diverse AI workloads. Job Scheduling: Prioritizes and queues jobs based on defined policies, ensuring fair usage of resources. 3. Use Case Use case: Training Large Language Models with LLaMA-factory One practical application of Slurm on Metal Cloud is training large language models using the LLaMA-factory framework. By distributing training across multiple GPUs and nodes, Slurm helps reduce training time while ensuring stable and efficient execution. Key Benefits: Scalability: Supports large-scale models with efficient GPU utilization. Cost Optimization: Reduces cloud computing costs by minimizing idle time. Reliability: Automated job resumption and error handling enhance workflow robustness. 4. Prerequisites Before proceeding, ensure you have the following: 4.1. System Requirements Metal Cloud access with multiple GPU-equipped nodes Slurm job scheduler installed and configured NVIDIA CUDA (11.8+ recommended) installed on all nodes NCCL (NVIDIA Collective Communication Library) for multi-GPU communication Python 3.8+ installed on all nodes Torch with distributed training support High-performance Storage 4.2. Network & SSH Configuration To enable seamless multi-node training, ensure: Each node can SSH into other nodes without a password using an SSH key. Network interfaces allow high-speed inter-node communication (e.g., InfiniBand). NCCL and PyTorch distributed backend can communicate over TCP/IP. You can verify node connectivity using: scontrol show nodes or sinfo 5. Environment Setup Assuming that you have all the system requirements for distributed training task, run the following on each compute node to install LLaMA-Factory. It will install all necessary packages to run LLaMA-Factory: [code lang="js"] python3 –m venv venv source venv/bin/activate git clone https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install –e “.[torch,metrics]” [/code] 6. Sample Training Task: Fine-Tuning LLaMA on Open Instruct Uncensored Alpaca Dataset To demonstrate a real-world scenario, we will full fine-tune a Llama-3.1-8B model using the Open Instruct Uncensored Alpaca dataset for instruction-following tasks. 6.1. Dataset: Open Instruct Uncensored Alpaca The Open Instruct Uncensored Alpaca is a collection of instruction-response pairs dataset. It is one of the most common datasets for fine-tuning models for instruction following. This dataset is public on Hugging Face. With LLaMA-Factory, you can specify the dataset's URI from a remote repository like Hugging Face directly in the YAML file to set up your training configuration, LLaMA-Factory will automatically download the dataset. To achieve this, you must define the dataset in a file named dataset_info.json, located in LLaMA-Factory/data/dataset_info.json. Add the following line to dataset_info.json. [code lang="js"] "uncensored_alpaca": {"hf_hub_url": "xzuyn/open-instruct-uncensored-alpaca"} [/code] When you have downloaded the dataset on the machine, you can add the following line to dataset_info.json and you are good to go. [code lang="js"]"your_dataset_name": {"file_name": "path/to/your/dataset.json"}[/code] 6.2. Model: LLaMA 3.1 8B The LLaMA 3.1 8B model is one of the latest releases in Meta’s third-generation LLaMA series. It is a lightweight yet powerful large language model designed for both research and enterprise applications. LLaMA 3.1 8B can be trained efficiently on multi-GPU multi-node Metal Cloud servers using LLaMA-Factory and DeepSpeed. The next sections of this guide will walk you through setting up distributed training for LLaMA 3.1 8B using LLaMA-Factory. If you want to download the model directly from Huggingface, you can use the below command: [code lang="js"] huggingface-cli download meta-llama/Llama-3.1-8B --local-dir=Llama-3.1-8B [/code] 7. Preparing training configuration LLaMA-Factory uses YAML configuration files to manage training parameters efficiently. A YAML-based configuration simplifies hyperparameter tuning and ensures reproducibility. This section explains how to prepare a YAML configuration file for fine-tuning the LLaMA 3.1 8B model using LLaMA-Factory. 7.1. Sample YAML Configuration for Fine-Tuning LLaMA 3.1 8B LLaMA-Factory provides various predefined YAML training configuration files, located at LLaMA-Factory/examples. Here is a YAML file for full fine-tuning LLaMA 3.1 8B with Open Instruct Uncensored Alpaca dataset: [code lang="js"] model_name_or_path: meta-llama/Llama-3.1-8B trust_remote_code: true stage: sft do_train: true finetuning_type: full deepspeed: examples/deepspeed/ds_z2_config.json dataset: uncensored_alpaca template: llama3 cutoff_len: 2048 max_samples: 500000 overwrite_cache: true preprocessing_num_workers: 16 output_dir: saves/llama3.1-8b/full/sft logging_steps: 10 save_steps: 10000 plot_loss: true overwrite_output_dir: true per_device_train_batch_size: 4 gradient_accumulation_steps: 2 learning_rate: 1.0e-5 num_train_epochs: 2.5 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ddp_timeout: 180000000 val_size: 0.001 per_device_eval_batch_size: 1 eval_strategy: steps eval_steps: 10000 [/code] You can put the model URI on Huggingface directly with: [code lang="js"] model_name_or_path: meta-llama/Llama-3.1-8B [/code] and LLaMA-Factory will automatically download the model before training. If you have downloaded the model on the machine, you can specify as: [code lang="js"] model_name_or_path: path/to/your/model [/code] To train on Open Instruct Uncensored Alpaca dataset, add the data by the specified name: [code lang="js"] dataset: uncensored_alpaca [/code] We adjust the number of training samples to 500,000 samples by: [code lang="js"] max_samples: 500000 [/code] You can adjust all other options if necessary 8. Configuring Slurm for Multi-Node Training Assume you have a YAML training configuration file named llama31_training.yaml, create a Slurm script train_llama.sbatch for training on 4 nodes, 8 GPUs per node: [code lang="js"] #!/bin/bash #SBATCH --job-name=multinode-training #SBATCH --nodes=4 #SBATCH --time=2-00:00:00 #SBATCH --gres=gpu:8 #SBATCH -o training.out #SBATCH -e training.err #SBATCH --ntasks=4 nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST ) ) nodes_array=($nodes) head_node=${nodes_array[0]} node_id=${SLURM_NODEID} head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address | cut -d" " -f2) echo Master Node IP: $head_node_ip export LOGLEVEL=INFO export NNODES=4 export NPROC_PER_NODE=8 export HEAD_NODE_IP=$head_node_ip export HEAD_NODE_PORT=29401 export NODE_RANK=$node_id export NCCL_IB_DISABLE=0 export NCCL_SOCKET_IFNAME=^lo,docker0 export NCCL_TIMEOUT=180000000 export NCCL_DEBUG=INFO export NCCL_BLOCKING_WAIT=1 # Ensure NCCL waits for operations to finish export NCCL_ASYNC_ERROR_HANDLING=1 # Allow handling of NCCL errors asynchronously source venv/bin/activate&amp;amp;amp;amp;amp;lt;/em&amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;lt;/wp-p&amp;amp;amp;amp;amp;gt; srun llamafactory-cli train llama31_training.yaml [/code] Use sbatch to submit the training job: sbatch train_llama.sbatch View the queue job with squeue Inspect the training.out and training.err file to see the training progress All 4 nodes are utilized perfectly. The final result is shown below: 9. Monitoring CPU and GPU Usage During Training When training large-scale models like LLaMA 3.1 8B on Metal Cloud, it is important to monitor system resources such as CPU, GPU, memory, and disk usage. Proper monitoring helps in: Detecting bottlenecks (e.g., underutilized GPUs, CPU overload). Optimizing resource allocation (e.g., adjusting batch sizes). Avoiding system crashes due to out-of-memory (OOM) errors. Bare Metal provides a monitoring page where users can track real-time hardware usage, including: GPU Utilization – See how much each GPU is being used. VRAM Usage – Check memory consumption per GPU. CPU Load – Monitor processor usage across nodes. Disk & Network Stats – Identify I/O bottlenecks. Users can access the monitoring page via their Metal Cloud dashboard to ensure efficient and stable training. Conclusion This guide provides a structured approach to setting up distributed training for LLaMA-Factory on Metal Cloud. We covered: Environment setup Slurm job submission Distributed training with LLaMA-Factory and DeepSpeed Optimizations for large-scale models Following these steps, you can fine-tune LLaMA models efficiently on Metal Cloud multi-node GPU clusters. Metal Cloud is now available on FPT AI Factory for reservation. Find out more at: https://aifactory.fptcloud.com/  For more information and consultancy, please contact: Hotline: 1900 638 399 Email: support@fptcloud.com Support: m.me/fptsmartcloud

Japan, November 13, 2024 — FPT, a global leading IT firm and Preferred NVIDIA Cloud Partner (NCP), officially announced the launch of the FPT AI Factory in Japan using the full-stack NVIDIA accelerated computing platform. This flagship solution serves as a one-stop shop for AI and Cloud services, offering immense computing power for AI advancement and contributing to developing Sovereign AI in the country. Japanese customers can expedite AI development with priority access to premium solutions and features through an exclusive pre-order.  The launching of FPT AI Factory in Japan by Mr. Le Hong Viet, CEO, FPT Smart Cloud, FPT Corporation At the NVIDIA AI Summit in Japan, FPT has debuted FPT AI Factory, an all-inclusive stack for end-to-end AI product lifecycle, including three main groups. FPT AI Infrastructure offers GPU cloud services with unprecedented computing power to accelerate model development and deployment. FPT AI Studio offers intelligent tools for building, pre-training, and fine-tuning AI models in depth using NVIDIA NeMo. FPT AI Inference, supported by NVIDIA NIM and NVIDIA AI Blueprints, enables customers to effectively deploy and scale their models in terms of size and number of usages. FPT AI Factory is integrated with 20+ ready-to-use AI products, built upon Generative AI, for rapid AI adoption and instant results in elevating customer experience, achieving operational excellence, transforming the human workforce, and optimizing operating expenses.    Powered by thousands of NVIDIA Hopper GPUs and next-generation ones, boosting with the latest NVIDIA AI Enterprise software platform, FPT AI Factory grants regional clientele scalable and confidential supercomputing along with essential tools to cultivate sophisticated AI technologies from scratch with faster time-to-market. This also empowers businesses to manage resources and processes expeditiously, optimizing total cost of ownership (TCO).   FPT is now accepting pre-orders for FPT AI Factory, allowing local corporate clients to leverage the diverse ecosystem of AI and Cloud, earn cloud credit, and gain early access to premium features. Combined with tailor-made consultation from seasoned AI & Cloud experts, enterprises in any industry can reinforce successful AI journeys with practical, high-value solutions.  Japan currently faces a shortage of GPU cloud solutions necessary to drive economic growth, promote innovation, and facilitate digital transformation in a secure manner. FPT’s AI-ready infrastructure in Japan provides local businesses and the government with unparalleled computational performance, high efficiency, and low-latency interactions to enrich research & development capabilities while safeguarding sensitive data and maintaining sovereignty.   By joining the NVIDIA Partner Network as a Service Delivery Partner, FPT will utilize NVIDIA's cutting-edge products and technologies to develop bespoke cloud services, hardware, software, and comprehensive integration services to drive the digital transformation in Japan.  Dr. Truong Gia Binh, FPT Corporation Chairman and Founder, shares FPT’s commitment to accompanying Japan in facilitating a successful AI journey Dr. Truong Gia Binh, FPT Corporation Chairman and Founder, affirmed, "Artificial intelligence continues to be a transformative technology for the entire world. In line with NVIDIA's global initiative, we are working closely with strategic partners to develop cloud infrastructure essential for AI applications worldwide, especially in Japan. We are committed to dedicating all necessary resources and accompanying the Japanese government, enterprises, and partners in the nation’s AI investment and development efforts. Through this significant project, we are aligning our vision and action to rapidly expand AI applications on a global scale while actualizing the collective vision of Japan and Vietnam in becoming AI nations.”  John Fanelli, NVIDIA Vice President of Enterprise AI Software, said, "In today's rapidly evolving technological landscape, Japan recognizes the importance of sovereign AI solutions for driving innovation, supporting data security, and maintaining technological independence. The FPT AI Factory built on NVIDIA accelerated computing and software represents a significant step towards meeting this need, offering Japanese companies access to cutting-edge AI infrastructure while fostering local AI development and expertise."  In April 2024, FPT publicized the development of AI Factories through a comprehensive strategic collaboration with NVIDIA. That marked a significant milestone in FPT's AI journey, aiming to promote AI research and development in the region and expand advanced AI and cloud capabilities on a global scale.        About FPT Corporation  FPT Corporation (FPT) is a global leading technology and IT services provider headquartered in Vietnam. FPT operates in three core sectors: Technology, Telecommunications, and Education. As AI is indeed a key focus, FPT has been integrating AI across its products and solutions to drive innovation and enhance user experiences within its Made by FPT ecosystem. FPT is actively working on expanding its capabilities in AI through investments in human resources, R&D, and partnerships with leading organizations like NVIDIA, Mila, AITOMATIC, and Landing AI. These efforts are aligned with FPT's ambitious goal to reach 5 billion USD in IT services revenue from global markets by 2030 and solidify its status among the world's top billion-dollar IT companies.  After nearly two decades in Japan, FPT has become one of the largest foreign-invested technology firms in the country by human resource capacity. The company delivers services and solutions to over 450 clients globally, with over 3,500 employees across 17 local offices and innovation hubs in Japan, and nearly 15,000 professionals supporting this market worldwide.   With Japan as a strategic focus for the company’s global growth, FPT has been actively expanding its business and engaging in M&A deals, such as the joint venture with Konica Minolta, strategic investment in LTS Inc, and most recently, the acquisition of NAC—its first M&A deal in the market. As digital transformation, particularly legacy system modernization viewed as a key growth driver in the Japanese market, the company is committed to providing end-to-end solutions and seamless services, utilizing advanced AI technologies as a primary accelerator. For more information, please visit https://fpt.com/en. 

Numerous reports from research institutions such as Gartner, Forrester, Bain, and Deloitte have predicted trending technology innovations for 2021. This article will introduce the top 10 trending technologies in those reports.  10 - Zero trust Security is one of the core focuses for businesses, and the zero-trust approach (no default security option) will get more prominent in the next few years. Gartner predicts that the calculation and security platforms can be categorized to 3 domains: Platform providing a safe environment to process or analyze personal and sensitive identification data  Platform allowing non-focused data process and analysis  Platform offering encryptions and algorithms prior to data processing or analysis (i.e.: synchronous encryption) 9 - Digital office and remote working Gartner points out that “working everywhere” is a crucial trend, especially under COVID-19-alike circumstances. Technologies within these domains will get more popular: Collaboration and productivity increase Secure remote access Edge computing Digital experience quantifiers Forrester foresees a business shift to local operations, in which technology will allow organizations to expand to new geographical areas.  8 - Human-centric digital experience The UI/UX and human-centered design technologies will be a core domain. Virtual Reality and Augmented Virtual Reality will be used more and more to enhance the experience.  The multi-experience development platform and low-code development platform will continue to grow, helping to build interactive applications and products as parts of the digital experience with multiple touchpoints for users (i.e.: touch, voice, and gestures).  7 - Super automation Super automation is the application of advanced technology (i.e.: AI, machine learning) to optimize human resources and automate processes, generating more impacts than the traditional automation approach.  Procedure and IT automation using AI, machine learning, event-driven architecture, and robotic process automation will remain an important movement. Prominent platforms: UIPath, Blue Prism,...  6 - Internet of Behaviors (IoB) The Internet of Behaviors (IoB) was introduced by Gote Nyman in 2012. IoB itself possesses social and moral influences that must be clearly understood.  The technology uses digital data obtained from the Internet of Things (IoT) devices to change human behaviors. For example, satellite data for driving behaviors, social media data to analyze trends, authentication data for security, and more. 5 - Intelligent business processes and platforms from the provider The agile organizational process to adapt correspondingly to the market or environment conditions. The Covid-19 pandemic has provoked a surge in this trend to enable existing technology and strategic support processes flexibly.  This includes faster decision-making, better supportive tech platforms, and more flexible providers to create possible business capabilities. 4 - 5G 5G promises to reduce delays by 100 times and provide information to construct new solutions.  Higher bandwidth for data sharing, higher transmission speed, and lower delays will certainly enhance the customer experience when connecting devices.  3 - Cybersecurity With increasingly remote activities, cybersecurity is a must for business. Gartner determines that cybersecurity will be a core technology in 2021. The security discovery will continue to be a significant movement to protect human identity and devices. 2 - Distributed Cloud Public Cloud vendors acknowledge the importance of supporting on-demand computing and edge computing. Yet, the continuous demand has tremendously stimulated distributed cloud computing. According to Gartner, the distributed cloud distributes public cloud services to different real locations while the initial public cloud provider is responsible for service operations, administrations, and updates.  1 - AI Technology AI technology is on top of the list because of its ability to enhance business continuity values. Organizations will optimize the advantages of AI through AI-powered operations using DataOps, ModelOps (MLOps), and DevOps. The use of AI on the Edge platform will increase, allowing AI algorithms to work on the edge of the network (closer to information gathering devices). Dominant platforms: Databricks, Snowflake, Azure Machine Learning, Amazon Sagemaker... The journey will continue and these top-notch technologies will further improve in 2021 and more. ———————————————————————— FPT Smart Cloud is committed to providing best-in-class products and services at the optimal expense for customers. Contact us now for support: Email: support@fptcloud.com Hotline: 1900 63 83 99