sia.hackernoon.com

Imagine looking for a specific type of clothing, not by describing it, but by showing a picture of something that looks like it. Or finding a hard-to-find bird among an enormous group of pictures, by simply submitting a photo. We're moving beyond keywords and leveraging the power of visual resemblance, opening up a universe of possibilities for how we interact with images.

In today's world of big data, images surround us everywhere. Traditionally used keyword-based search is often not sufficient, as it cannot capture the richness of visual content. That is where image embeddings and vector search come into play. These methods are extremely powerful and are revolutionizing the way we search and analyze images, enabling us to search for the unreachable depths and get important insights from visual data.

How Does Vector Search Work?

So, we have these image fingerprints. What’s next? How do we use them for searching? That is where vector search comes in. When we search for a picture, the search engine first generates an embedding for our query picture. Next, it matches the query vector against all the pictures in the database and their embeddings. It calculates the "distance" between the vectors, and the more alike the vectors are, the more alike the pictures are visually.

This distance calculation is a fundamental part of vector search. There can be many different metrics for distance that can be used, but the idea is that less distance equals more similarity. The search engine brings back the images with the closest vectors to the query vector, which is basically finding the most visually similar images. It is similar to placing locations on a map; images with similar visual characteristics will cluster together.

Real-World Applications

There are many and varied applications of image embeddings and vector search:

E-commerce: Think about buying a new sofa. Instead of trying to type on the website what type of style we are seeking, we can post a picture of a similar sofa we are interested in. Vector search by the e-commerce website finds sofas it has on hand that closely resemble what we have uploaded, and makes our shopping much simpler.
Content Moderation: Flagging offending or offensive images within user-uploaded content is a continuing issue. Vector search can be utilized to accomplish this automatically by comparing uploaded images to a database of known offending images and flagging up potentially offending content for human review.
Medical Imaging: Doctors can look for comparable cases in a medical image database through vector search. They can diagnose rare diseases or find proper treatments based on their previous experience.
Image Retrieval: Think about searching through a large database of old photographs. Vector search enables us to search for images even if they have no descriptive tags.

Example of how we can achieve this in SQL in BigQuery:

Imagine you're turning the pages of a magazine, one filled with images of grand, elegant homes, or in modern times, scrolling online. Your heart is set on a particular colonial-style house. You can see it: the symmetrical facade, the stately columns, perhaps of a specific color. You're on a quest to find that perfect dwelling, the one that matches the very picture.

And, you want to use the image you have to find similar homes. Hence, in this example, we would use the image of a colonial-style home that you have and search across other homes we have.

Basically, you're using that picture as a reference point to look through other available homes, trying to find ones that share the same characteristics. It's like saying, "Show me more of this."

We have a set of images of homes for demo purposes. These images are generated using Imagen on Vertex AI.

And in order to find the home we want, we would be using image embedding in BigQuery and the Vector Search functionality of BigQuery.

The first step would be to create a BigQueryML model.

CREATE OR REPLACE MODEL `[PROJECT_ID.DATASET_ID.MODEL_NAME]`
REMOTE WITH CONNECTION `[PROJECT_ID.REGION.CONNECTION_ID]`
OPTIONS (ENDPOINT = 'multimodalembedding@001');

Next, we need to create an object table over all the images of the homes in the inventory, on top of which we want to search.

CREATE OR REPLACE EXTERNAL TABLE `[PROJECT_ID.DATASET_ID].external_images_table`
WITH CONNECTION `[PROJECT_ID.REGION.CONNECTION_ID]`
OPTIONS(
object_metadata = 'SIMPLE',
uris = ['[BUCKET_NAME]'],
max_staleness = INTERVAL 1 DAY,
metadata_cache_mode = 'AUTOMATIC');

Now, we will generate the embeddings of the images stored in the Google Cloud Storage bucket using the ML.GENERATE_EMBEDDING function.

CREATE OR REPLACE TABLE `[PROJECT_ID.DATASET_ID].home_embeddings` AS
SELECT *
FROM ML.GENERATE_EMBEDDING(
 MODEL `[PROJECT_ID.DATASET_ID.MODEL_NAME]`,
 TABLE `[PROJECT_ID.DATASET_ID].external_images_table`,
 STRUCT(TRUE AS flatten_json_output,
 512 AS output_dimensionality)
);


SELECT * FROM `[PROJECT_ID.DATASET_ID].house_embeddings`;

Now, we will search using the VECTOR_SEARCH() function to search for similar homes like the one we want.

But first, we would have to generate the embeddings of the image that we want to search for.



-- Create External Table for the test image


CREATE OR REPLACE EXTERNAL TABLE `[PROJECT_ID.DATASET_ID].external_images_test_table`
WITH CONNECTION `[PROJECT_ID.REGION.CONNECTION_ID]`
OPTIONS(
object_metadata = 'SIMPLE',
uris = ['[BUCKET_NAME]'],
max_staleness = INTERVAL 1 DAY,
metadata_cache_mode = 'AUTOMATIC');


-- Generate Embeddings of the test image


CREATE OR REPLACE TABLE `[PROJECT_ID.REGION].test_embeddings` AS
SELECT *
FROM ML.GENERATE_EMBEDDING(
 MODEL `[PROJECT_ID.REGION.MODEL_NAME]`,
 TABLE `[PROJECT_ID.DATASET_ID].external_images_test_table`,
 STRUCT(TRUE AS flatten_json_output,
 512 AS output_dimensionality)
);

Vector Search

SELECT base.uri AS image_link, distance
FROM
VECTOR_SEARCH(
 TABLE  `[PROJECT_ID.DATASET_ID].home_embeddings`,
 'ml_generate_embedding_result',
 (
 SELECT * FROM `[PROJECT_ID.REGION].test_embeddings`
 ),
 top_k => 5,
 distance_type => 'COSINE',
 options => '{"use_brute_force":true}'
);

Result

It is very interesting to see the search results. We notice that the top results returned are very similar to the Colonial style we were hoping to find. They were a close match to our initial image, suggesting the search is working well.

This enables faster and more scalable semantic search by using an approximate nearest neighbor search algorithm.

Conclusion

Image embeddings and vector search are changing how we interact with visual data. They help us search past keywords, allowing us to use the true capabilities of image search and power many forms of novel applications. When these technologies further evolve, we can expect progressively sophisticated and easy-to-use mechanisms of searching, analyzing, and understanding the world of vision. The future is about visuals, and it's being fueled by vectors.