> ## Documentation Index
> Fetch the complete documentation index at: https://docs.radicalwhale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Importing Data

> Import data into your datasets

## Import Methods

<CardGroup cols={2}>
  <Card title="File Upload" icon="upload">
    CSV, Excel, JSON files up to 100MB
  </Card>

  <Card title="API Integration" icon="plug">
    Real-time data import via REST API
  </Card>

  <Card title="Database Connection" icon="database">
    PostgreSQL, MySQL, SQL Server, Oracle, MongoDB
  </Card>

  <Card title="Third-Party" icon="link">
    Google Sheets, Airtable, webhooks
  </Card>
</CardGroup>

## File Upload

Navigate to your dataset → **Import Data** → **File Upload**.

### Supported Formats

**CSV**: Best for structured data. UTF-8 encoding recommended, 100MB limit.

**Excel**: .xlsx/.xls files, multiple sheets supported, formulas converted to values.

**JSON**: Array of objects or line-delimited JSON, automatic schema detection.

### Upload Process

1. **Select File**: Browse or drag-and-drop your file
2. **Configure**: System detects headers, delimiters, and encoding automatically
3. **Map Columns**: Match file columns to dataset columns, create new columns if needed
4. **Handle Issues**: System flags missing headers, type mismatches, duplicates, invalid dates
5. **Choose Import Mode**:
   * **Append**: Add to existing data
   * **Replace**: Replace all data
   * **Update**: Update existing records by key column
6. **Execute Import**: Review summary and start processing

### Best Practices

**File Preparation**: Use clean headers, consistent formatting, UTF-8 encoding, remove empty rows.

**Large Files**: Split files over 100MB, remove unnecessary columns, test with sample first.

## API Integration

Use the REST API for real-time or automated imports.

### Setup

1. **Get API Key**: Workspace Settings → API Keys → Create with `datasets:write` permission
2. **Use Endpoint**: `POST /api/v1/workspaces/{workspace_id}/datasets/{dataset_id}/records`
3. **Authentication**: Bearer token in Authorization header

### Example

```python theme={null}
import requests
import json

# Configuration
api_key = "your_api_key_here"
workspace_id = "your_workspace_id"
dataset_id = "your_dataset_id"
base_url = "https://api.radicalwhale.com"

# Headers
headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

# Data to import
data = [
    {
        "company_name": "Example Corp",
        "website": "https://example.com",
        "industry": "Technology"
    }
]

# Make the request
url = f"{base_url}/api/v1/workspaces/{workspace_id}/datasets/{dataset_id}/records"
response = requests.post(url, headers=headers, json=data)

if response.status_code == 201:
    result = response.json()
    print(f"Successfully imported {result['imported_count']} records")
else:
    print(f"Import failed: {response.status_code} - {response.text}")
```

**Batch Processing**: Use 100-1000 records per batch for large datasets.

**Validation**: Include `validate_schema`, `skip_duplicates`, `update_existing` in options.

## Database Connections

Workspace Settings → Integrations → **Add Database Connection**.

**Supported**: PostgreSQL (9.6+), MySQL (5.7+), SQL Server (2017+), Oracle (12c+), MongoDB (4.0+)

**Setup**: Enter host, port, database name, credentials, and SSL settings. Test connection before saving.

**Import**: Select tables/views, map columns, add filtering with WHERE clauses, schedule automatic syncs.

**Custom SQL**: Use custom queries for advanced data selection and transformation.

### Best Practices

**Performance**: Use indexes, limit results with WHERE clauses, schedule during off-hours, use incremental updates.

**Security**: Read-only database accounts, SSL/TLS connections, IP whitelisting.

## Third-Party Integrations

### Google Sheets

Connect Google account → Select spreadsheet → Configure range → Set sync schedule.

**Tips**: Use headers in first row, consistent data types, avoid merged cells, remove empty rows.

### Airtable

Enter Airtable API key → Select base and tables → Map fields → Configure attachment and linked record handling.

### Webhooks

Get webhook endpoint URL → Configure security tokens → Define payload format → Test with sample data.

Real-time import with automatic validation, error logging, and rate limiting.

## Next Steps

<CardGroup cols={2}>
  <Card title="Managing Columns" icon="columns" href="/guides/datasets/managing-columns">
    Configure and optimize dataset columns
  </Card>

  <Card title="Working with Records" icon="table" href="/guides/datasets/working-with-records">
    Edit, filter, and manage imported data
  </Card>

  <Card title="Creating Agents" icon="robot" href="/guides/agents/creating-agents">
    Set up AI agents to process imported data
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/datasets/import">
    Technical documentation for import APIs
  </Card>
</CardGroup>