NVIDIA Triton Inference Server |
What is an Inference Server?
The role of an Inference Server is to accept user input data and pass it to an underlying trained model in the required format and return the results. It is also widely known as a Prediction Server as the results are nothing but predictions (in most cases).
The NVIDIA Triton server is a gold standard that standardizes AI model deployment and execution across every workload and it is important to know how it works internally for your custom or off the shelf models.