Building Machine Learning Powered Applications: Going from Idea to Product
Last read July 31, 2024
View on Amazon
Highlights
15 highlights.
Server-side deployment consists of setting up a web server that can accept requests from clients, run them through an inference pipeline, and return the results.
Location: 3,721
Streaming workflows accept requests as they come and process them immediately.
Location: 3,725
Batch workflows are run less frequently and process a large number of requests all at once.
Location: 3,726
Figure 9-1. Streaming API workflow
Location: 3,738
The batch approach considers the inference pipeline as a job that can be run on multiple examples at once. A batch job runs a model on many examples and stores predictions so they can be used when needed. Batch jobs are appropriate when you have access to the features needed for a model before the model’s prediction is required.
Location: 3,776
Figure 9-3. Example of batch workflow
Location: 3,791
Figure 9-4. A model running inference on device (we can still train on a server)
Location: 3,810
Figure 9-5. Running on a server, or locally
Location: 3,821
In neural networks, for example, weights are often pruned (removing those with values close to zero) and quantized (lowering the precision of weights).
Location: 3,828
The right approach depends on your application’s needs, such as latency requirements, hardware, network and privacy concerns, and inference costs.
Location: 3,894
Figure 10-2. Example branching logic for input checks
Location: 3,946
The worse model makes more mistakes, but its mistakes are different from the complex model because of the different shape of its decision boundary. Because of this, the simpler model gets some examples right that the complex model gets wrong.
Location: 3,984
This is the intuition for why using a simple model as a backup is a reasonable idea when a primary model fails.
Location: 3,986
If those probabilities are well calibrated (see “Calibration Curve”), they can be used to detect instances where a model is uncertain and decide not to display results to a user.
Location: 4,003
Figure 10-8. Deploying an updated version of the same model can seem like a simple change
Location: 4,137