How to read CSV files in TensorFlow?

November 19, 2024

Learn how to efficiently read and process CSV files in TensorFlow with this comprehensive guide. Ideal for data scientists and AI enthusiasts.

How to read CSV files in TensorFlow?

Import Required Libraries

First, ensure that you have TensorFlow installed. If not, install it using pip:

pip install tensorflow

Import necessary modules from TensorFlow and other packages:

import tensorflow as tf

Load CSV Files Using tf.data API

The `tf.data` API provides functions like `tf.data.experimental.make_csv_dataset` for loading CSV files efficiently. This is especially useful for large datasets.

# Define the file path and parameters
filename = 'your_data.csv'
batch_size = 32  # Process data in batches

# Load the CSV as a dataset
dataset = tf.data.experimental.make_csv_dataset(
    filename,
    batch_size=batch_size,
    label_name='target_column',  # specify the label/target column
    na_value="?",
    num_epochs=1,
    ignore_errors=True
)

The function automatically infers the schema for the data, handles missing values, and shuffles the dataset for you if required.

Inspect the Dataset

You can iterate over the dataset to inspect its structure or access individual batches:

for batch in dataset.take(1):   # Examine a single batch
    features, labels = batch
    print("Features: ", features)
    print("Labels: ", labels)

This outputs a dictionary of features and a tensor of labels for you to examine.

Preprocess the Data

Preprocessing is often necessary. Use map functions to apply transformations to each element of the dataset.

def preprocess(features, label):
    # Example: Normalize a feature
    features['feature_name'] = features['feature_name'] / 100.0
    return features, label

# Apply the preprocessing function
dataset = dataset.map(preprocess)

Adjust the preprocessing function to suit your specific needs such as normalization, feature extraction, or handling missing data.

Integrate with TensorFlow Models

Once your data is loaded and preprocessed, you can integrate it with TensorFlow models. Here's an example with a simple Sequential model:

# Define a simple model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='relu', input_shape=(len(features),)),  # adjust input_shape
    tf.keras.layers.Dense(1)
])

# Compile the model
model.compile(optimizer='adam', loss='mean_squared_error')

# Train the model using the dataset
model.fit(dataset, epochs=10)

Make sure to provide an appropriate input shape and adjust the model to suit your specific problem.

Reading Multiple CSV Files

If you have multiple CSV files, you can use wildcards in the file path and load them into a single dataset:

file_pattern = 'data/*.csv'  # Adjust path as needed

# Load multiple CSV files
dataset = tf.data.experimental.make_csv_dataset(
    file_pattern,
    batch_size=batch_size,
    label_name='target_column',
    na_value="?",
    num_epochs=1,
    ignore_errors=True
)