Data Import

The data importer is a great tool for importing data from external sources. It can be used for the initial setup of a project or continuous synchronization with a third-party application.

Currently, the data importer supports products, prices, price groups, and currencies.

What the Data Importer Does

The data importer inserts data that have not been previously imported into the Ucommerce database. If an entity already exists, it updates the existing entity. The only exception is the currency entity; if a currency with the same ISO code already exists in the database, that entity will not be updated.

Getting Started

To use the data importer, install the Ucommerce.DataImport.Core NuGet package:

dotnet add package Ucommerce.DataImport.Core

After installing the package, implement the following interfaces to fetch data from the external source(s):

• ICurrencyFetcher

• IPriceGroupFetcher

• IPriceFetcher

• IProductPriceFetcher

• IProductFetcher

Each interface has a method to fetch the data. The method takes batchNumber and batchSize as parameters. Use these to fetch a specific amount of data on each call. For example, if batchNumber is 2 and batchSize is 100, the fetcher should skip the first 200 elements and take the next 100:

var elements = data.Skip(batchNumber * batchSize).Take(batchSize);

Below is an example of how CurrencyFetcher could be implemented by reading from a CSV file:

public class MyCurrencyFetcher : ICurrencyFetcher
{
    public virtual Task<IReadOnlyList<CurrencyData>> GetCurrencyData(int batchNumber, int batchSize)
    {
            var filePath = "currencies.csv";
            using var reader = new StreamReader(filePath);
            using var csv = new CsvReader(reader, config);
            csv.Context.RegisterClassMap(new CurrencyMap());
            var records = csv.GetRecords<CurrencyData>();
            records = records.Skip(batchNumber * batchSize)
                .Take(batchSize);
            return Task.FromResult<IReadOnlyList<CurrencyData>>(records.ToImmutableList());
    }
}

Registering the Fetcher(s)

The fetcher(s) need to be registered in the service collection. It's recommended to create a helper method to register all the fetchers for reusability:

public static class MyPIMExtensions
{
    public static IServiceCollection AddPIMFetchers(this IServiceCollection services)
    {
        services.AddSingleton<ICurrencyFetcher, MyCurrencyFetcher>();
        services.AddSingleton<IProductFetcher, MyProductFetcher>();
        ...
        return services;
    }
}

Setting up the Data Importer

Add the necessary settings to the appsettings.json file (or similar) under the Ucommerce section to set up the data importer. The two key settings are BatchSize (optional - default is 1000), which determines the size of data chunks to be fetched, and ConnectionString (required), which should point to the Ucommerce database the data will be imported into. Here is an example of the app settings for the data importer:

{
  "Ucommerce": {
    "DataImport": {
      "BatchSize": 100,
      "ConnectionString": "YourConnectionString"
    }
  }
}

Running the Data Importer

To run the data importer, inject it where you want to execute it and call the Run method. Here is an example of an API Controller with an endpoint for executing the data importer on demand:

public class DataImportController : Controller
{
    private readonly DataImporter _dataImporter;

    public FeatureTestingController(DataImporter dataImporter)
    {
        _dataImporter = dataImporter;
    }

    public virtual async Task<IActionResult> ImportData()
    {
        await _dataImporter.Run();
        return Ok();
    }
}

It is generally recommended to execute the importer on a schedule, e.g. as a background service as part of the website, or as a console application with access to both the external source and the Ucommerce database outside the website.