Replicate img2text

This handler was implemented using the replicate library that is provided by Replicate.

The required arguments to establish a connection are,

• model_name: Model name which you want to access in MindsDB. e.g ‘air-forever/kandinsky-2’
• version: version hash/id which you want to use in MindsDB.
• api_key: API key from Replicate Platform you can found here.

Before you can use Replicate, it’s essential to authenticate by setting your API token in an environment variable named REPLICATE_API_TOKEN. This token acts as a key to enable access to Replicate’s features.

1. Using pip:

If you’re working in a standard Python environment (using pip for package management), set your token as an environment variable by running the following command in your terminal:

On Linux, Mac:

export REPLICATE_API_TOKEN='YOUR_TOKEN'

On Windows:

set REPLICATE_API_TOKEN=YOUR_TOKEN

2. Using Docker:

For Docker users, the process slightly differs. You need to pass the environment variable directly to the Docker container when running it. Use this command:

docker run -e REPLICATE_API_TOKEN='YOUR_TOKEN' -p 47334:47334 -p 47335:47335 mindsdb/mindsdb

Again, replace ‘YOUR_TOKEN’ with your actual Replicate API token.

Usage

To use this handler and connect to a Replicate cluster in MindsDB, you need an account on Replicate. Make sure to create an account by following this link.

To establish the connection and create a model in MindsDB, use the following syntax:

CREATE MODEL blip

PREDICT text

USING

    engine = 'replicate',

    model_name= 'salesforce/blip',

    version ='2e1dddc8621f72155f24cf2e0adbde548458d3cab9f00c0139eea840d0ac4746',

    api_key = 'r8_BpO.........................';

You can use the DESCRIBE PREDICTOR query to see the available parameters that you can specify to customize your predictions:

DESCRIBE PREDICTOR mindsdb.blip.features;

OUTPUT

+----------+------------------+-----------------------------------------------------------------------+--------+

| inputs   | default          | description                                                           | type   |

+----------+------------------+-----------------------------------------------------------------------+--------+

| task     | image_captioning | Choose a task.                                                        | -      |

| image    | -                | Input image                                                           | string |

| caption  | -                | Type caption for the input image for image text matching task.        | string |

| question | -                | Type question for the input image for visual question answering task. | string |

+----------+------------------+-----------------------------------------------------------------------+--------+

Visual Question Answering

Now, you can use the established connection to query your ML Model as follows:

SELECT *

FROM mindsdb.blip

WHERE image="https://images.unsplash.com/photo-1686847266385-a32745169de4"

AND question="Is there lion in image?"

USING 

task="visual_question_answering";

OUTPUT

+------------+--------------------------------------------------------------+-------------------------+

| text        | image                                                        | question                |

+------------+--------------------------------------------------------------+-------------------------+

| Answer: no | https://images.unsplash.com/photo-1686847266385-a32745169de4 | Is there lion in image? |

+------------+--------------------------------------------------------------+-------------------------+

Image Captioning

SELECT *

FROM mindsdb.blip

WHERE image="https://images.unsplash.com/photo-1686847266385-a32745169de4"

OUTPUT

+---------------------------------------------------+--------------------------------------------------------------+

| text                                               | image                                                        |

+---------------------------------------------------+--------------------------------------------------------------+

| Caption: a bird is sitting on the back of a horse | https://images.unsplash.com/photo-1686847266385-a32745169de4 |

+---------------------------------------------------+--------------------------------------------------------------+

Image Text Matching

SELECT *

FROM mindsdb.blip

WHERE image="https://images.unsplash.com/photo-1686847266385-a32745169de4"

AND caption="Bird having horse Riding"

USING 

task="image_text_matching";

OUTPUT

+-----------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+--------------------------+

| text                                                                                                                               | image                                                        | caption                  |

+-----------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+--------------------------+

| The image and text is matched with a probability of 0.7730.

The image feature and text feature has a cosine similarity of 0.3615. | https://images.unsplash.com/photo-1686847266385-a32745169de4 | Bird having horse Riding |

+-----------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+--------------------------+

This is just an one model used in this example there are more with vast variation and usescases. Also there is no limit to imagination, how can you use this.

Note: Replicate provides only a few free predictions, so choose your predictions wisely. Don’t let the machines have all the fun, save some for yourself! 😉