[{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/06-TESTING-AND-CONSUMPTION-PLAN.html","title":"06 - Testing, Consumption, and F# Language Integration Plan\n","content":"06 - Testing, Consumption, and F# Language Integration Plan\nThis document details the test strategy, consumption validation, performance benchmarking, and F#-specific language features for Parquet.TypeProvider.\n\n1. F#-Bespoke Language Behaviors\nF# data engineers and developers expect idioms distinct from C# object-oriented paradigms. Parquet.TypeProvider natively integrates with the F# type system:\nmindmap\n  root((F# Parquet Behaviors))\n    Nullability\n      ::icon(fa fa-check)\n      \u0027T option default\n      \u0027T voption support\n      No nulls in F# domain\n    Units of Measure\n      int64\u0026amp;lt;ms\u0026amp;gt; for timestamps\n      decimal\u0026amp;lt;USD\u0026amp;gt; for monetary fields\n      float\u0026amp;lt;m\u0026amp;gt; for coordinates\n    Structural Semantics\n      IStructuralEquatable\n      IComparable\n      Set / Map / distinct compatibility\n    Active Patterns\n      Partial patterns for enums\n      Pattern matching on schemas\n    Execution Pipelines\n      Seq.map / filter / fold\n      taskSeq / AsyncSeq streaming\n\nA. Idiomatic Option Handling (\u0027T option \u0026amp; \u0027T voption)\n\nParquet OPTIONAL Columns: Automatically mapped to F# Some(\u0027value) or None.\nZero-allocation voption: For tight loops, users can specify PreferStructOption = true to emit ValueOption\u0026lt;\u0027T\u0026gt; to avoid heap allocations.\n\nB. Units of Measure Support\n\n\nTimestamp Precision: Timestamps and duration fields can be tagged with measure units:\nfsharp\n[\u0026lt;Measure\u0026gt;] type ms\n[\u0026lt;Measure\u0026gt;] type us\n[\u0026lt;Measure\u0026gt;] type ns\n[\u0026lt;Measure\u0026gt;] type USD\n\nCompile-Time Type Safety: Prevents accidental mathematical operations mixing milliseconds with seconds or mismatched currencies.\n\nC. Structural Equality \u0026amp; Comparisons\n\nGenerated erased row instances implement IStructuralEquatable and IComparable.\nRows can be directly placed into F# Set, used as keys in Map, or deduplicated using Seq.distinct.\n\nD. Pipeline \u0026amp; Functional Combinators\n\n\nDirect integration with Seq, Array, and List modules:\nfsharp\nlet highValueOrders =\nOrders.Load(\u0022orders.parquet\u0022)\n|\u0026gt; Seq.filter (fun o -\u0026gt; o.Amount \u0026gt; 1000.0m\u0026lt;USD\u0026gt;)\n|\u0026gt; Seq.groupBy (fun o -\u0026gt; o.CustomerId)\n|\u0026gt; Seq.map (fun (cid, group) -\u0026gt; cid, Seq.sumBy (fun o -\u0026gt; o.Amount) group)\n\n\n\n2. Multi-Tier Testing Machinery\nflowchart TD\n    subgraph Layer1 [1. Schema \u0026amp; Design-Time Tests]\n        T1[Sample File Resolution]\n        T2[Thrift / FileMetaData Parser]\n        T3[ProvidedTypeDefinition Verification]\n    end\n\n    subgraph Layer2 [2. Binary Data \u0026amp; Cross-Engine Parity]\n        T4[Parquet.Net v6 Generated Files]\n        T5[PyArrow / Pandas Generated Files]\n        T6[DuckDB Generated Files]\n        T7[ParquetSharp C\u002B\u002B Generated Files]\n    end\n\n    subgraph Layer3 [3. Memory \u0026amp; Streaming Tests]\n        T8[10M Row Stream Memory Ceiling]\n        T9[Row Group Disposal \u0026amp; GC Verification]\n    end\n\n    subgraph Layer4 [4. Package Consumption Tests]\n        T10[Local NuGet Package Build]\n        T11[Consumer F# Project Integration]\n        T12[F# Interactive .fsx Out-of-Process Test]\n    end\n\nLayer 1: Schema \u0026amp; Provided Type Unit Tests (tests/Parquet.TypeProvider.Tests)\n\nValidates all 13 core Parquet physical and logical types (Boolean, Int32, Int64, Float, Double, Decimal, Guid, String, Byte[], Date, Time, Timestamp, List).\nVerifies compiler diagnostic reporting for missing files or corrupted footers.\n\nLayer 2: Cross-Engine Binary Parity Tests\n\n\nTests ingestion against standard files produced by:\n\nPython pyarrow / fastparquet\nDuckDB\nApache Spark\nParquetSharp (C\u002B\u002B Arrow)\n\n\n\nLayer 3: Memory \u0026amp; Resource Leak Tests\n\nStreams a 10-million row dataset across 1,000 row groups.\nAsserts that memory remains constant (GC.GetTotalMemory(true) delta \u0026lt; 50MB) proving row groups are streamed and discarded rather than retained.\n\n\n3. End-to-End Package Consumption Testing\nTo ensure the packaged NuGet asset functions seamlessly in consumer applications and developer tooling:\nA. Local Packaging Pipeline\n\nOutput .nupkg into local ./artifacts directory via dotnet pack.\n\nConfigure test/Parquet.TypeProvider.ConsumerApp with a local nuget.config:\nxml\n\u0026lt;configuration\u0026gt;\n\u0026lt;packageSources\u0026gt;\n\u0026lt;add key=\u0022local-artifacts\u0022 value=\u0022../../artifacts\u0022 /\u0026gt;\n\u0026lt;/packageSources\u0026gt;\n\u0026lt;/configuration\u0026gt;\n\n\nB. IDE \u0026amp; Tooling Verification Matrix\n\nF# Interactive (dotnet fsi): Script #r \u0022nuget: Parquet.TypeProvider, 0.0.1\u0022 executes cleanly.\nRider / Ionide / Visual Studio: Verifies type completion, parameter info, and tooltips load without design-time host lockups.\n\n\n4. Multi-Scale Benchmarking Suite\nLocated in benchmarks/Parquet.TypeProvider.Benchmarks utilizing BenchmarkDotNet:\nMulti-Scale Testing (1K, 10K, 100K, 1,000,000 Rows)\ngantt\n    title Benchmark Comparison Workloads\n    dateFormat X\n    axisFormat %s\n    section 1K Rows (Micro-latency)\n    Baseline vs TypeProvider : 0, 1000\n    section 100K Rows (ETL Batch)\n    Memory \u0026amp; Throughput : 0, 100000\n    section 1M Rows (High-Throughput)\n    Streaming Execution : 0, 1000000\n\nBenchmarked Candidates:\n\n*ParquetSerializer.DeserializeAsync* (Parquet.Net Reflection Baseline).\n*Parquet.FSharp* (F# Runtime Reflection Mapper).\nManual ParquetReader Loop (Hand-optimized raw C# / F# baseline).\n*Parquet.TypeProvider seq\u0026lt;Row\u0026gt;* (Idiomatic row traversal).\n*Parquet.TypeProvider \u0060Columns.\u0060** (Direct columnar array extraction).\n\nMeasured Metrics:\n\nMean Execution Time (ms / \u03BCs).\nMemory Allocated per Operation (MB / KB).\nGen 0, Gen 1, Gen 2 GC Collections.\n\n\n5. Planned Sample Applications\n\n*samples/Exploration.fsx*: Interactive analytics notebook script.\n*samples/StreamingETL/*: Multi-gigabyte streaming pipeline using IAsyncEnumerable.\n*samples/UnitsOfMeasure/*: Financial/telemetry modeling with strongly-typed units (\u0026lt;ms\u0026gt;, \u0026lt;USD\u0026gt;).\n\n","headings":["06 - Testing, Consumption, and F# Language Integration Plan","1. F#-Bespoke Language Behaviors","B. Units of Measure Support","C. Structural Equality \u0026amp; Comparisons","D. Pipeline \u0026amp; Functional Combinators","2. Multi-Tier Testing Machinery","Layer 2: Cross-Engine Binary Parity Tests","Layer 3: Memory \u0026amp; Resource Leak Tests","3. End-to-End Package Consumption Testing","A. Local Packaging Pipeline","B. IDE \u0026amp; Tooling Verification Matrix","4. Multi-Scale Benchmarking Suite","Multi-Scale Testing (1K, 10K, 100K, 1,000,000 Rows)","Benchmarked Candidates:","Measured Metrics:","5. Planned Sample Applications"],"type":"content"},{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/03-TYPE-MAPPINGS.html","title":"03 - Parquet to F# Type Mappings\n","content":"03 - Parquet to F# Type Mappings\nThis document defines how Apache Parquet physical and logical types are mapped to the F# type system.\n\n1. Primitive \u0026amp; Scalar Mappings\n\n\n\nParquet Physical / Logical Type\nRequired Column (F#)\nOptional Column (PreferOption=true)\n\n\n\n\nBOOLEAN\nbool\nbool option\n\n\nINT32\nint32\nint32 option\n\n\nINT64\nint64\nint64 option\n\n\nINT96 (Legacy Timestamp)\nSystem.DateTime\nSystem.DateTime option\n\n\nFLOAT\nfloat32\nfloat32 option\n\n\nDOUBLE\nfloat\nfloat option\n\n\nBYTE_ARRAY (String / UTF8)\nstring\nstring option\n\n\nBYTE_ARRAY (Raw Binary)\nbyte[]\nbyte[] option\n\n\nFIXED_LEN_BYTE_ARRAY (Guid)\nSystem.Guid\nSystem.Guid option\n\n\nDECIMAL\ndecimal\ndecimal option\n\n\nTIMESTAMP_MILLIS / MICROS\nSystem.DateTime\nSystem.DateTime option\n\n\nDATE\nSystem.DateOnly / System.DateTime\nSystem.DateOnly option\n\n\nTIME_MILLIS / MICROS\nSystem.TimeSpan\nSystem.TimeSpan option\n\n\n\n\n\n2. Nullability \u0026amp; Option Handling\nParquet fields declare repetition levels:\n- REQUIRED: Mapped directly to the non-nullable F# type \u0027T.\n- OPTIONAL:\n- When PreferOption = true (default): Mapped to \u0027T option.\n- When PreferOption = false: Value types mapped to Nullable\u0026lt;\u0027T\u0026gt;, reference types mapped to nullable references.\n\n3. Nested Structures \u0026amp; Collections (Phase 3)\n\n\n\nParquet Structure\nF# Representation\n\n\n\n\nLIST\u0026lt;T\u0026gt;\n\u0027T list / \u0027T[]\n\n\nMAP\u0026lt;K, V\u0026gt;\nMap\u0026lt;\u0027K, \u0027V\u0026gt; / IDictionary\u0026lt;\u0027K, \u0027V\u0026gt;\n\n\nSTRUCT\nGenerated Nested Provided Type (Parent.NestedStruct)\n\n\n\n\n","headings":["03 - Parquet to F# Type Mappings","1. Primitive \u0026amp; Scalar Mappings","2. Nullability \u0026amp; Option Handling","3. Nested Structures \u0026amp; Collections (Phase 3)"],"type":"content"},{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/02-ROADMAP.html","title":"02 - Roadmap \u0026amp; Milestones\n","content":"02 - Roadmap \u0026amp; Milestones\nPhase 1: Foundation \u0026amp; Design-Time Schema Inference (Milestone 1)\n\n[ ] Initialize repository structure with FSharp.TypeProviders.SDK.\n[ ] Implement sample file loading and resolution (relative paths, absolute paths).\n[ ] Implement Parquet schema inspection from Parquet.Net metadata.\n[ ] Generate basic ProvidedTypeDefinition types with scalar properties (primitives, strings, decimals).\n[ ] Wire basic quotation expressions for typed row instantiation.\n\nPhase 2: Runtime Engine \u0026amp; Columnar Execution (Milestone 2)\n\n[ ] Implement ParquetRowContext columnar batch representation.\n[ ] Implement synchronous .Load(filePath) and .Load(stream) row enumeration.\n[ ] Support option\u0026lt;\u0027T\u0026gt; for optional / nullable Parquet columns.\n[ ] Implement Parquet.TypeProvider.Tests covering all core Parquet primitive data types.\n\nPhase 3: Advanced Types \u0026amp; Streaming (Milestone 3)\n\n[ ] Async \u0026amp; streaming APIs: .AsyncLoad(...) and IAsyncEnumerable\u0026lt;\u0027Row\u0026gt;.\n\n[ ] Complex type support:\n\nTimestamp units (Millis, Micros).\nEnums and custom logical types.\nLists and nested structures.\n\n\n[ ] Static column accessor overloads (columnar arrays without creating row objects).\n\nPhase 4: Integration Testing, CI, \u0026amp; Packaging (Milestone 4)\n\n[ ] Multi-targeting and multi-IDE testing (Visual Studio, Rider, VS Code / Ionide).\n[ ] F# Interactive (.fsx) and Polyglot Notebook verification.\n[ ] Performance benchmarking vs naive reflection readers (BenchmarkDotNet).\n[ ] GitHub Actions CI workflow with NuGet packaging and SourceLink.\n\n","headings":["02 - Roadmap \u0026amp; Milestones","Phase 1: Foundation \u0026amp; Design-Time Schema Inference (Milestone 1)","Phase 2: Runtime Engine \u0026amp; Columnar Execution (Milestone 2)","Phase 3: Advanced Types \u0026amp; Streaming (Milestone 3)","Phase 4: Integration Testing, CI, \u0026amp; Packaging (Milestone 4)"],"type":"content"},{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/07-ECOSYSTEM-AND-DEPENDENCIES.html","title":"07 - F# Software Foundation Ecosystem \u0026amp; Dependency Standards\n","content":"07 - F# Software Foundation Ecosystem \u0026amp; Dependency Standards\nThis document establishes the architectural standards, language baselines, and rationale for external dependencies in Parquet.TypeProvider, specifically emphasizing the role of the F# Software Foundation (FSSF) and the fsprojects community ecosystem.\n\n1. Language Baseline: F# 6.0\u002B\nParquet.TypeProvider establishes F# 6.0 (shipped with .NET 6 / FSharp.Core \u0026gt;= 6.0.0) as the minimum language baseline.\nRationale for F# 6.0 Baseline:\n\n\nNative task { ... } Computation Expressions:\n\nStarting in F# 6.0, task { ... } is built directly into the core language compiler via resumable code.\nGenerates high-performance, low-allocation struct state machines without requiring legacy computation expression shims (e.g., TaskBuilder.fs or Ply).\n\n\n\nDirect Interoperability with .NET Asynchronous APIs:\n\nParquet.Net v6 low-level readers (ParquetReader.CreateAsync, groupReader.ReadAsync) return standard .NET Task\u0026lt;\u0027T\u0026gt;, which are awaited with native let! / do! in F# 6\u002B with zero overhead.\n\n\n\n\n2. The F# Software Foundation (fsprojects) Governance\nThe F# Software Foundation is the independent, non-profit organization dedicated to advancing the F# language and ecosystem. Projects hosted under the fsprojects GitHub organization adhere to specific community governance standards:\n\nLong-Term Maintainability: Not tied to single-author abandonment; shared maintainer access across the core F# community.\nFirst-Class Compiler Compatibility: Built to evolve in lockstep with FSharp.Core and the F# Compiler Service (FCS).\nIdiomatic Language Design: Built specifically for F# idioms (algebraic data types, computation expressions, immutability, pipeline operators).\n\n\n3. Technical Evaluation of Foundation Dependencies\nFSharp.Control.TaskSeq (fsprojects/FSharp.Control.TaskSeq)\n\nRole in Parquet.TypeProvider: Powers the non-blocking asynchronous streaming engine (readRowsStream / loadFromFileAsync / AsyncLoad).\n\nWhy it was selected:\n\nThe Missing Piece in FSharp.Core: While F# 6.0 introduced native task { ... } for scalar tasks (Task\u0026lt;\u0027T\u0026gt;), it did not introduce a native taskSeq builder for asynchronous streams (IAsyncEnumerable\u0026lt;\u0027T\u0026gt;).\nZero OS Thread Blocking: Completely eliminates the anti-pattern of calling .GetAwaiter().GetResult() inside sequence expressions.\nStandard Interface: Produces and consumes standard .NET IAsyncEnumerable\u0026lt;\u0027T\u0026gt;, ensuring seamless interop with ASP.NET Core streaming, channels, and cloud SDKs.\n\n\n\nFSharp.TypeProviders.SDK (fsprojects/FSharp.TypeProviders.SDK)\n\nRole in Parquet.TypeProvider: Provides the type generation and compiler infrastructure for Parquet.TypeProvider.DesignTime.\n\nWhy it was selected:\n\nThe official, canonical framework maintained by the F# community for creating generative and erased Type Providers.\nManages design-time vs. runtime assembly separation, caching, and quotation tree generation.\n\n\n\n\n4. Dependency Selection Matrix \u0026amp; Policy\nTo guarantee enterprise stability and prevent dependency bloat, all dependencies must meet these criteria:\nflowchart TD\n    Candidate[Third-Party Dependency Candidate] --\u0026gt; CheckFSSF{Hosted under fsprojects or .NET Foundation?}\n    CheckFSSF -- Yes --\u0026gt; VerifyAOT{Zero trim/AOT warnings?}\n    CheckFSSF -- No --\u0026gt; CheckMaint{Multi-maintainer \u0026amp; active CI?}\n    CheckMaint -- Yes --\u0026gt; VerifyAOT\n    CheckMaint -- No --\u0026gt; Reject[Reject or Vendor Minimal Implementation]\n    VerifyAOT -- Yes --\u0026gt; Approve[Approved for Dependency Inclusion]\n    VerifyAOT -- No --\u0026gt; Reject\n\n\n\n\nDependency\nCategory\nGovernance\nPurpose\n\n\n\n\n*Parquet.Net*\nCore Engine\nManaged .NET / AloneGuid\nLow-level Parquet format chunk reader \u0026amp; writer\n\n\n*FSharp.TypeProviders.SDK*\nDesign-Time SDK\nF# Software Foundation (fsprojects)\nType Provider compiler infrastructure\n\n\n*FSharp.Control.TaskSeq*\nAsync Streaming\nF# Software Foundation (fsprojects)\ntaskSeq { ... } computation expressions\n\n\n*BenchmarkDotNet*\nBenchmarks\n.NET Foundation\nMulti-scale performance baseline verification\n\n\n*xUnit*\nTest Framework\n.NET Foundation\nAutomated unit and integration testing\n\n\n\n\n\n5. Guidelines for Future F# Extension Packages\nWhen extending Parquet.TypeProvider (e.g., adding railway-oriented error handling, cancellable tasks, or property testing), priority is given to:\n1. Error Handling: FsToolkit.ErrorHandling (taskResult { ... }, taskOption { ... }).\n2. Cancellable Async: IcedTasks (cancellableTask { ... }).\n3. Property Testing: FsCheck (fsprojects).\n","headings":["07 - F# Software Foundation Ecosystem \u0026amp; Dependency Standards","1. Language Baseline: F# 6.0\u002B","Rationale for F# 6.0 Baseline:","3. Technical Evaluation of Foundation Dependencies","4. Dependency Selection Matrix \u0026amp; Policy","5. Guidelines for Future F# Extension Packages"],"type":"content"},{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/01-ARCHITECTURE-PLAN.html","title":"01 - Architecture Plan: Parquet.TypeProvider\n","content":"01 - Architecture Plan: Parquet.TypeProvider\n1. Overview\nParquet.TypeProvider is an F# Type Provider that generates strongly-typed representations of Apache Parquet schemas at compile time. It enables F# developers to seamlessly read and query Parquet datasets with full static type checking, auto-completion, and optimized execution.\nflowchart LR\n    subgraph DesignTime [Design Time / Compilation]\n        Sample[Sample .parquet File / Schema] --\u0026gt; SchemaParser[Schema Inference Engine]\n        SchemaParser --\u0026gt; TPGenerator[ProvidedTypeDefinition Generator]\n        TPGenerator --\u0026gt; FSCompiler[F# Compiler / IDE IntelliSense]\n    end\n\n    subgraph Runtime [Execution Time]\n        DataFile[Production Parquet Stream / File] --\u0026gt; ColumnDecoder[Low-Level Parquet.Net Chunk Reader]\n        ColumnDecoder --\u0026gt; RowInstantiator[Generated Row Accessors / Seq]\n        RowInstantiator --\u0026gt; Consumer[User F# Code]\n    end\n\n\n2. Type Provider SDK Architecture\nFollowing F# Type Provider best practices and the FSharp.TypeProviders.SDK, the project is cleanly divided into two assemblies:\nA. Design-Time Component (Parquet.TypeProvider.DesignTime)\n\nRole: Runs inside the IDE (Visual Studio, Rider, VS Code / Ionide) and the F# compiler (fsc.exe).\n\nResponsibilities:\n\nLocates the sample Parquet file or schema specification from the static parameter.\nParses the Parquet file metadata (FileMetaData, Schema, Thrift schema fields) using Parquet.Net metadata readers.\nUses ProvidedTypeDefinition, ProvidedProperty, and ProvidedMethod to generate the erased types.\nEmits runtime quotation expressions (\u0026lt;@@ ... @@\u0026gt;) that wire property getters directly to underlying row index arrays.\n\n\n\nB. Runtime Component (Parquet.TypeProvider.Runtime)\n\nRole: Deployed with the application and referenced at runtime.\n\nResponsibilities:\n\nParquetReaderCore: Wraps ParquetReader and ParquetRowGroupReader from Parquet.Net.\nParquetRowContext: Lightweight column storage representing decoded row-group batches in memory.\nParquetSequence: Implements seq\u0026lt;\u0027Row\u0026gt; and IAsyncEnumerable\u0026lt;\u0027Row\u0026gt; over batched row groups.\n\n\n\n\n3. Memory \u0026amp; Performance Strategy\n\n\nErased Types:\n\n\nTypes are erased at runtime to avoid code bloat. Row objects are represented internally by a lightweight context or struct wrapping column arrays:\nfsharp\ntype ParquetRowContext = {\nRowIndex: int\nColumns: obj[]\n}\n\n\n\n\nColumnar Batching over Row-by-Row Decoding:\n\nInstead of reading row-by-row with reflection, Parquet.Net reads an entire column chunk into a contiguous typed array (int[], string[], DateTime[], etc.).\n\nThe generated row property accessor simply performs an array index lookup:\nfsharp\n// Property getter expression emitted by design-time provider:\n\u0026lt;@@ fun (ctx: ParquetRowContext) -\u0026gt; (ctx.Columns.[columnIdx] :?\u0026gt; \u0027FieldType[])[ctx.RowIndex] @@\u0026gt;\n\n\n\n\nRow-Group Streaming:\n\nReading does not require buffering the entire file. When iterating via seq or IAsyncEnumerable, row groups are loaded, processed, and released sequentially.\n\n\n\n\n4. Key Static Parameters\nThe type provider accepts the following static parameters:\ntype Dataset = ParquetProvider\u0026lt;\n    Sample = \u0026quot;path/to/sample.parquet\u0026quot;,      // Local path or URL to sample file\n    Schema = \u0026quot;\u0026quot;,                            // Optional inline schema definition\n    PreferOption = true,                    // Map nullable columns to \u0026#39;T option (vs null)\n    BatchSize = 10000                       // Default row group batch size\n\u0026gt;\n\n","headings":["01 - Architecture Plan: Parquet.TypeProvider","1. Overview","2. Type Provider SDK Architecture","3. Memory \u0026amp; Performance Strategy","4. Key Static Parameters"],"type":"content"},{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/05-TESTING-AND-BENCHMARKS.html","title":"05 - Testing Strategy \u0026amp; Benchmarking System\n","content":"05 - Testing Strategy \u0026amp; Benchmarking System\nThis document outlines the testing machinery and performance benchmark suite for Parquet.TypeProvider, incorporating best practices and benchmark automation from Parquet.SourceGenerator.\n\n1. Testing Machinery\nBecause an F# Type Provider generates types inside the compiler host, testing requires three distinct layers:\nflowchart TD\n    subgraph Unit [1. Unit \u0026amp; Schema Tests]\n        SchemaTests[Schema Parser \u0026amp; Metadata Inspection]\n        TypeMapTests[ProvidedType Signature Validation]\n    end\n\n    subgraph Roundtrip [2. Binary Data Roundtrip]\n        ParquetNetData[Parquet.Net Standard Output] --\u0026gt; TPReader[TypeProvider .Load]\n        TPReader --\u0026gt; ParquetSharpData[ParquetSharp C\u002B\u002B Cross-Validation]\n    end\n\n    subgraph Harness [3. Host \u0026amp; Script Harness]\n        FSXScripts[F# Interactive .fsx Evaluation]\n        Polyglot[Polyglot / Jupyter Notebook Scenarios]\n    end\n\nA. Design-Time Provided Type Verification\nUsing FSharp.TypeProviders.SDK test helpers to assert that ProvidedTypeDefinition and generated properties match the inferred Parquet schema without executing runtime loads.\nB. Binary Compatibility \u0026amp; Cross-Validation\n\nReading test parquet files generated by Parquet.Net v6.\nReading parquet files generated by Python pyarrow / pandas and C\u002B\u002B ParquetSharp to ensure full specification compliance (timestamps, nullability, decimals, strings).\n\nC. F# Interactive (.fsx) Script Testing\nEnsuring that #r \u0022nuget: ...\u0022 loads cleanly out-of-process in FSI (F# Interactive), Rider, Visual Studio, and Ionide.\n\n2. Benchmarking Architecture (BenchmarkDotNet)\nBenchmarking is located in benchmarks/Parquet.TypeProvider.Benchmarks and models the multi-scale methodology of Parquet.SourceGenerator:\nMulti-Scale Workloads\n\nSmall Batches: 1,000 rows (latency and per-record overhead).\nMedium Batches: 10,000 to 100,000 rows (ETL and memory GC pressure).\nLarge Batches: 1,000,000 rows (streaming row group throughput).\n\nBenchmark Comparison Matrix\n\n\n\nBenchmark Scenario\nBaseline 1 (Reflection)\nBaseline 2 (F# Mapper)\nTarget: Parquet.TypeProvider\n\n\n\n\nRow Iteration (seq\u0026lt;\u0027Row\u0026gt;)\nParquetSerializer.DeserializeAsync\nParquet.FSharp (Reflection)\nDirect columnar array indexing\n\n\nDirect Column Array Read\nN/A\nManual ParquetReader loop\nOrders.Columns.Amount(...)\n\n\nMemory Allocation\nBaseline\nBaseline\nZero-copy ArrayPool chunking\n\n\n\n\nBenchmark Automation \u0026amp; CI Reports\nBenchmarks are configured with [MemoryDiagnoser] and [Orderer(SummaryOrderPolicy.FastestToSlowest)]. CI runs update the benchmark markdown summary block in the README.md automatically via GitHub Actions.\n","headings":["05 - Testing Strategy \u0026amp; Benchmarking System","1. Testing Machinery","A. Design-Time Provided Type Verification","B. Binary Compatibility \u0026amp; Cross-Validation","Multi-Scale Workloads","Benchmark Comparison Matrix","Benchmark Automation \u0026amp; CI Reports"],"type":"content"},{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/04-API-DESIGN.html","title":"04 - API Design \u0026amp; Usage Patterns\n","content":"04 - API Design \u0026amp; Usage Patterns\nThis document describes the public surface area and common usage patterns for Parquet.TypeProvider.\n\n1. Type Provider Declaration\nopen Parquet.TypeProvider\n\n// Type definition with sample file\ntype Orders = ParquetProvider\u0026lt;\u0026quot;../data/samples/orders.parquet\u0026quot;\u0026gt;\n\n\n2. Reading Data\nA. Synchronous File / Stream Loading\n// From local path\nlet data: seq\u0026lt;Orders.Row\u0026gt; = Orders.Load(\u0026quot;path/to/orders_2026.parquet\u0026quot;)\n\n// From open stream\nuse fileStream = File.OpenRead(\u0026quot;path/to/orders_2026.parquet\u0026quot;)\nlet dataFromStream = Orders.Load(fileStream)\n\nfor row in data do\n    printfn $\u0026quot;Order ID: {row.OrderId}, Total: {row.Amount}\u0026quot;\n\nB. Async / Streaming Loading\ntask {\n    let! rows = Orders.AsyncLoad(\u0026quot;https://storage.blob.core.windows.net/data/orders.parquet\u0026quot;)\n    for row in rows do\n        // Process rows asynchronously\n        do! processRowAsync row\n}\n\nC. Direct Columnar Access (Zero-Allocation Batch Reading)\nFor high-performance analytical scenarios that don\u0027t need row abstractions:\n// Read columns directly as contiguous typed arrays\nlet orderIds: int64[] = Orders.Columns.OrderId(\u0026quot;path/to/orders.parquet\u0026quot;)\nlet amounts: decimal[] = Orders.Columns.Amount(\u0026quot;path/to/orders.parquet\u0026quot;)\n\n\n3. Interactive Data Exploration (F# Interactive / Notebooks)\nIn .fsx or Jupyter / Polyglot Notebooks:\n#r \u0026quot;nuget: Parquet.TypeProvider\u0026quot;\n\nopen Parquet.TypeProvider\n\ntype Trades = ParquetProvider\u0026lt;\u0026quot;trades_sample.parquet\u0026quot;\u0026gt;\nlet df = Trades.Load(\u0026quot;trades_sample.parquet\u0026quot;)\n\ndf\n|\u0026gt; Seq.filter (fun t -\u0026gt; t.Symbol = \u0026quot;AAPL\u0026quot;)\n|\u0026gt; Seq.averageBy (fun t -\u0026gt; float t.Price)\n|\u0026gt; printfn \u0026quot;Average AAPL Price: %f\u0026quot;\n\n","headings":["04 - API Design \u0026amp; Usage Patterns","1. Type Provider Declaration","2. Reading Data","A. Synchronous File / Stream Loading","B. Async / Streaming Loading","C. Direct Columnar Access (Zero-Allocation Batch Reading)","3. Interactive Data Exploration (F# Interactive / Notebooks)"],"type":"content"},{"uri":"https://github.com/rtkelly13/Parquet.TypeProvider/INDEX.html","title":"Parquet.TypeProvider Documentation Index\n","content":"Parquet.TypeProvider Documentation Index\nWelcome to the technical documentation for Parquet.TypeProvider.\nDocumentation Map\n\n\n\uD83D\uDCD0 01 - Architecture \u0026amp; Design\n\nHigh-level architecture, design-time vs runtime assemblies, type erasure, and memory model.\n\n\n\n\uD83D\uDDFA\uFE0F 02 - Roadmap \u0026amp; Milestones\n\nPhase-by-phase implementation plan from initial schema parsing to production release.\n\n\n\n\uD83D\uDD20 03 - Type Mappings Specification\n\nComprehensive matrix of Parquet physical and logical types mapped to F# types and option\u0026lt;\u0027T\u0026gt;.\n\n\n\n\uD83D\uDEE0\uFE0F 04 - API Design \u0026amp; Usage Patterns\n\nSurface area for synchronous, asynchronous, streaming, and direct column array access.\n\n\n\n\u26A1 05 - Testing Strategy \u0026amp; Benchmarks\n\nMulti-scale BenchmarkDotNet suite, binary compatibility verification, and CI automation.\n\n\n\n\uD83E\uDDEA 06 - Testing, Consumption, \u0026amp; F# Language Integration\n\nIn-depth plan covering full test coverage, local package consumer testing, F#-bespoke type behaviors, and samples.\n\n\n\n\uD83C\uDFDB\uFE0F 07 - F# Software Foundation Ecosystem \u0026amp; Dependencies\n\nArchitectural standards, F# Software Foundation (fsprojects) governance, and technical rationale for FSharp.Control.TaskSeq and FSharp.TypeProviders.SDK.\n\n\n\n","headings":["Parquet.TypeProvider Documentation Index","Documentation Map"],"type":"content"}]